time=2026-10-20T00:14:03.097Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-20T00:14:03.476Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-20T00:14:03.476Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-20T00:14:03.887Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-20T00:14:04.156Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-20T00:14:04.156Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-20T00:14:04.247Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-20T00:15:01.574Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-20T00:15:01.574Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc80b6d518e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc80b6d518e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xbabb28f39e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-zk8hdsmg8dfw" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "name": "7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "status": "Failed",
      "startTime": "2026-10-20T00:39:28.091453996Z",
      "endTime": "2026-10-20T00:58:30.991397546Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
            correlation ID: 8265cd70-ed50-4525-a687-45c965024ae6
            client request ID: b95e9e3d-a0f7-4ee8-8ac7-2ff5d62a1d3a
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xbabb28f39e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-zk8hdsmg8dfw" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "name": "7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "status": "Failed",
      "startTime": "2026-10-20T00:39:28.091453996Z",
      "endTime": "2026-10-20T00:58:30.991397546Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
            correlation ID: 8265cd70-ed50-4525-a687-45c965024ae6
            client request ID: b95e9e3d-a0f7-4ee8-8ac7-2ff5d62a1d3a

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1fe322c7e500>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2qqsdjg4hsrq" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "name": "7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "status": "Failed",
      "startTime": "2026-10-20T00:36:56.91702797Z",
      "endTime": "2026-10-20T00:55:57.103083611Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh to exist\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-gl748: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-gl748 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-2qqsdjg4hsrq--managed/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Network/networkInterfaces/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to htt... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
            correlation ID: 2083ceae-e353-4ba6-a1e3-ff92ee472926
            client request ID: 9c9e7813-8848-4608-b832-b1dbe65ca4ad
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1fe322c7e500>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2qqsdjg4hsrq" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "name": "7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "status": "Failed",
      "startTime": "2026-10-20T00:36:56.91702797Z",
      "endTime": "2026-10-20T00:55:57.103083611Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh to exist\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-gl748: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-gl748 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-2qqsdjg4hsrq--managed/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Network/networkInterfaces/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to htt... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
            correlation ID: 2083ceae-e353-4ba6-a1e3-ff92ee472926
            client request ID: 9c9e7813-8848-4608-b832-b1dbe65ca4ad
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xaeb08df4660>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xaeb08df4660>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cde6af4c2e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-19T23:40:50.5353725Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1cde6af4c2c0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1cde6a0b6140>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1cde6af4c2a0>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1cde6c048750>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cde6af4c2e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-19T23:40:50.5353725Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1cde6af4c2c0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1cde6a0b6140>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1cde6af4c2a0>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1cde6c048750>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x17293bf48060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-qw8hmnl947sz" to finish creating: GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd",
      "name": "2be64b2f-b78e-49f6-addd-04faf33378bd",
      "status": "Failed",
      "startTime": "2026-10-19T23:54:37.149407446Z",
      "endTime": "2026-10-20T00:13:41.899452027Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
            correlation ID: a4d6efb0-533f-4043-ad67-651fe433c379
            client request ID: 6f8a1c34-923e-40e5-a5b3-733c67d8d88f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x17293bf48060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-qw8hmnl947sz" to finish creating: GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd",
      "name": "2be64b2f-b78e-49f6-addd-04faf33378bd",
      "status": "Failed",
      "startTime": "2026-10-19T23:54:37.149407446Z",
      "endTime": "2026-10-20T00:13:41.899452027Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
            correlation ID: a4d6efb0-533f-4043-ad67-651fe433c379
            client request ID: 6f8a1c34-923e-40e5-a5b3-733c67d8d88f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.009s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x207ea69dfda0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.009s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x207ea69dfda0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-20T00:03:40Z
  Ended:       2026-10-20T00:18:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0435200-svc", component="backend", name="ingresscertificate", region="westus3", severity="warning"
  Description: Backend controller workqueue ingresscertificate has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-10-19T23:00:45.350Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-19T23:00:46.914Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-19T23:00:47.803Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f66b53e6663a909706c4cc56f884e08afa3f703badb52eda8425598d78f027f5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1281792%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff66b53e6663a909706c4cc56f884e08afa3f703badb52eda8425598d78f027f5
time=2026-10-19T23:08:19.085Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1281792/providers/Microsoft.Resources/deployments/f66b53e6663a909706c4cc56f884e08afa3f703badb52eda8425598d78f027f5/operationStatuses/08584091560383429326\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1281792/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1281792/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j1281792\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n                \\\"message\\\": \\\"Database account creation failed. Operation Id: 84ce1c4b-104f-4d59-ba97-ca7f687e5f6a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 50b97f78-d517-42ba-b6b3-2e3807c11c56, Request URI: /apps/56b56b41-1aa1-47de-9428-4a9b9ccbfcc7/services/5492004a-a82d-4653-81b2-013acecf0ba5/partitions/4d300830-d333-45bc-8ad6-8d4a8244255d/replicas/134308569966594579s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-10-19T23:06:28.2874662Z, RequestEndTime: 2026-10-19T23:06:33.6942116Z,  Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:05:35.8111686Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.582,\\\\\\\"memory\\\\\\\":21738860.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0143,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:05:45.8134136Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.599,\\\\\\\"memory\\\\\\\":21747896.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0589,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":27},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:05:55.8162151Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.543,\\\\\\\"memory\\\\\\\":21739028.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0371,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:06:05.8318538Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.956,\\\\\\\"memory\\\\\\\":21715984.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.3135,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:06:15.8338344Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.371,\\\\\\\"memory\\\\\\\":21721028.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0646,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:06:25.8342074Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.996,\\\\\\\"memory\\\\\\\":21715496.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0674,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32}]}\\\\r\\\\nRequestStart: 2026-10-19T23:06:28.2874662Z; ResponseTime: 2026-10-19T23:06:33.6942116Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-centralus1-be312.documents.azure.com:11303/apps/56b56b41-1aa1-47de-9428-4a9b9ccbfcc7/services/5492004a-a82d-4653-81b2-013acecf0ba5/partitions/4d300830-d333-45bc-8ad6-8d4a8244255d/replicas/134308569966594579s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5410.799, ActivityId: 50b97f78-d517-42ba-b6b3-2e3807c11c56, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11303 | status: Connected | lkt: 10/19/2026 11:06:33 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2874662Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0013},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2874675Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0046},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2874721Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0625},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2875346Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5411.5582},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:33.6990928Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0545},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:33.6991473Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-10-19T23:04:15.0012932Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-10-19T23:04:15.0012932Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-10-19T23:04:16.0332114Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":867,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x15ac4447a018>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x15ac43752020>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
                err: <*fmt.wrapError | 0x15ac44e58300>{
                    msg: "DNS for route host agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
                    err: <*fmt.wrapError | 0x15ac434e29a0>{
                        msg: "DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x15ac4447a018>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x15ac43752020>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
                err: <*fmt.wrapError | 0x15ac44e58300>{
                    msg: "DNS for route host agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
                    err: <*fmt.wrapError | 0x15ac434e29a0>{
                        msg: "DNS for agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rzzvn.apps.aro.cilium-cl.2ru7.j2965632.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1d0051281e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6h69n8f46csz" to finish creating: GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f",
      "name": "65c67213-a430-416b-a8e1-9c576702841f",
      "status": "Failed",
      "startTime": "2026-10-19T23:03:54.135232982Z",
      "endTime": "2026-10-19T23:22:56.513764235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
            correlation ID: d01ca859-c416-4ccc-bd23-51544b0af025
            client request ID: 937df210-9f80-46da-9e07-0166c55c864d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1d0051281e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6h69n8f46csz" to finish creating: GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f",
      "name": "65c67213-a430-416b-a8e1-9c576702841f",
      "status": "Failed",
      "startTime": "2026-10-19T23:03:54.135232982Z",
      "endTime": "2026-10-19T23:22:56.513764235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
            correlation ID: d01ca859-c416-4ccc-bd23-51544b0af025
            client request ID: 937df210-9f80-46da-9e07-0166c55c864d
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:44:19Z
  Ended:       2026-10-19T22:50:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4417280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti4iojordd6hm0nnh191a3fmg3e1aub-f4w6r1h1a4m1v8n", pod="openshift-apiserver-7dc759866b-hr75b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ti4iojordd6hm0nnh191a3fmg3e1aub-f4w6r1h1a4m1v8n/openshift-apiserver-7dc759866b-hr75b has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p"
Unexpected error:
    <*fmt.wrapError | 0x2d7273ae000>: 
    failed to create NodePool np-vkgl4p: failed waiting for nodepool="np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p" in resourcegroup="rg-hs-pre--j799lrwqxz6j" to finish creating: GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "name": "c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "status": "Failed",
      "startTime": "2026-10-19T22:21:51.863828648Z",
      "endTime": "2026-10-19T22:40:54.499105259Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: Failed: virtualmachine failed to create or update. err: failed to get existing resource rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
            correlation ID: 2e542bba-f131-4b5d-bf4c-a77a8bc95e2d
            client request ID: 83a4b938-e107-403c-83e6-0592ebdaeff2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p"
Unexpected error:
    <*fmt.wrapError | 0x2d7273ae000>: 
    failed to create NodePool np-vkgl4p: failed waiting for nodepool="np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p" in resourcegroup="rg-hs-pre--j799lrwqxz6j" to finish creating: GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "name": "c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "status": "Failed",
      "startTime": "2026-10-19T22:21:51.863828648Z",
      "endTime": "2026-10-19T22:40:54.499105259Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: Failed: virtualmachine failed to create or update. err: failed to get existing resource rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
            correlation ID: 2e542bba-f131-4b5d-bf4c-a77a8bc95e2d
            client request ID: 83a4b938-e107-403c-83e6-0592ebdaeff2
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-d9xldnlstk6j/cluster-candidate-5-0-tlxgzk should provision
Unexpected error:
    <*fmt.wrapError | 0x2c66751baf40>: 
    failed to create HCP cluster cluster-candidate-5-0-tlxgzk: failed waiting for cluster="cluster-candidate-5-0-tlxgzk" in resourcegroup="rg-candidate-5-0-d9xldnlstk6j" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065",
      "name": "6dc91de3-05e9-401b-9139-c466ece83065",
      "status": "Failed",
      "startTime": "2026-10-19T22:04:14.113898084Z",
      "endTime": "2026-10-19T22:33:23.409451279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
            correlation ID: 2a2d3ba7-babe-4fe8-ae4e-5435562f5656
            client request ID: a7fdb086-12e2-40d2-b194-107f4afa279d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-d9xldnlstk6j/cluster-candidate-5-0-tlxgzk should provision
Unexpected error:
    <*fmt.wrapError | 0x2c66751baf40>: 
    failed to create HCP cluster cluster-candidate-5-0-tlxgzk: failed waiting for cluster="cluster-candidate-5-0-tlxgzk" in resourcegroup="rg-candidate-5-0-d9xldnlstk6j" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065",
      "name": "6dc91de3-05e9-401b-9139-c466ece83065",
      "status": "Failed",
      "startTime": "2026-10-19T22:04:14.113898084Z",
      "endTime": "2026-10-19T22:33:23.409451279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
            correlation ID: 2a2d3ba7-babe-4fe8-ae4e-5435562f5656
            client request ID: a7fdb086-12e2-40d2-b194-107f4afa279d
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:180]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x222007964a08>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x222007b84d20>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x2220079649d8>{
                    errs: [
                        <*errors.errorString | 0x222008c33e60>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:180]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x222007964a08>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x222007b84d20>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x2220079649d8>{
                    errs: [
                        <*errors.errorString | 0x222008c33e60>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-24dvjs" for cluster "cluster-hs-pre-24dvjs"
Unexpected error:
    <*fmt.wrapError | 0x3d397c5fe960>: 
    failed to create NodePool np-24dvjs: failed waiting for nodepool="np-24dvjs" for cluster "cluster-hs-pre-24dvjs" in resourcegroup="rg-hs-pre--qmdx974n6w26" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "name": "8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "status": "Failed",
      "startTime": "2026-10-19T22:14:24.493403189Z",
      "endTime": "2026-10-19T22:33:34.182492247Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
            correlation ID: 33ce383b-ea0f-4d30-8902-6098f7bbfeb8
            client request ID: 9ce1f451-6c72-4c5f-8183-8c4490de781d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-24dvjs" for cluster "cluster-hs-pre-24dvjs"
Unexpected error:
    <*fmt.wrapError | 0x3d397c5fe960>: 
    failed to create NodePool np-24dvjs: failed waiting for nodepool="np-24dvjs" for cluster "cluster-hs-pre-24dvjs" in resourcegroup="rg-hs-pre--qmdx974n6w26" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "name": "8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "status": "Failed",
      "startTime": "2026-10-19T22:14:24.493403189Z",
      "endTime": "2026-10-19T22:33:34.182492247Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
            correlation ID: 33ce383b-ea0f-4d30-8902-6098f7bbfeb8
            client request ID: 9ce1f451-6c72-4c5f-8183-8c4490de781d
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:40:56Z
  Ended:       2026-10-19T22:52:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frhvn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:28:50Z
  Ended:       2026-10-19T22:55:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="43f3c967-dd75-47e1-aa51-7f18084ed726"
  Description: Pod ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T22:29:50Z
  Ended:       2026-10-19T22:55:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="43f3c967-dd75-47e1-aa51-7f18084ed726"
  Description: Pod ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:25:56Z
  Ended:       2026-10-19T22:51:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frhvn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T22:25:56Z
  Ended:       2026-10-19T22:51:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frhvn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 10 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T21:42:05Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-796b59578b-rbqnk", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-796b59578b-rbqnk has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T21:42:05Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-vr6v5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-vr6v5 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T21:42:05Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-9vsw9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-9vsw9 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3pc5270b8h60pst14u078uaplvhlq-q0m7n1d5p4o1u4j", pod="azure-disk-csi-driver-controller-65f5bf78b-s9gq4", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3pc5270b8h60pst14u078uaplvhlq-q0m7n1d5p4o1u4j/azure-disk-csi-driver-controller-65f5bf78b-s9gq4 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-sb8lp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-sb8lp has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-796b59578b-bjj96", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-796b59578b-bjj96 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-796b59578b-w4b2v", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-796b59578b-w4b2v has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-8qtrk", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-8qtrk has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-mkmmq", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-mkmmq has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-rvxrh", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-rvxrh has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xcdba5b46020>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-9bqdtwnxktt9" to finish creating: GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "name": "f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "status": "Failed",
      "startTime": "2026-10-19T21:20:53.14343965Z",
      "endTime": "2026-10-19T21:40:01.878429551Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
            correlation ID: 85a1b8fd-965d-4893-90c4-5bc86f87328d
            client request ID: 6c4ed168-8339-4ad9-9c7e-dcd04bbc9a76
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xcdba5b46020>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-9bqdtwnxktt9" to finish creating: GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "name": "f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "status": "Failed",
      "startTime": "2026-10-19T21:20:53.14343965Z",
      "endTime": "2026-10-19T21:40:01.878429551Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
            correlation ID: 85a1b8fd-965d-4893-90c4-5bc86f87328d
            client request ID: 6c4ed168-8339-4ad9-9c7e-dcd04bbc9a76

alert fired 13 time(s) (1 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T20:56:48Z
  Ended:       2026-10-19T21:03:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7537536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti35rfutnepdsfn0vfq6aa39v1svufp-f2i3r5n2t0v2v3v", pod="router-6dbb4f8499-ccrn9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti35rfutnepdsfn0vfq6aa39v1svufp-f2i3r5n2t0v2v3v/router-6dbb4f8499-ccrn9 has been in a non-ready state for longer than 5 minutes.

time=2026-10-19T20:14:27.366Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T20:14:27.872Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T20:14:27.872Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T20:14:28.027Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-19T20:14:28.232Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-19T20:14:28.232Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T20:14:28.292Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T20:15:24.446Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T20:15:24.446Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:437]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x340bb99d4c0>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rhzhzsd2dvs2" to finish creating: GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de",
      "name": "1520252c-b285-49f1-8706-c82f9e3a45de",
      "status": "Failed",
      "startTime": "2026-10-19T20:28:35.166007052Z",
      "endTime": "2026-10-19T20:47:38.665697527Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
            correlation ID: da09f0ee-1b1c-4dd0-986e-72ed6522e248
            client request ID: 773c74d7-3374-4a99-9619-c2776f1df3e0
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:437]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x340bb99d4c0>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rhzhzsd2dvs2" to finish creating: GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de",
      "name": "1520252c-b285-49f1-8706-c82f9e3a45de",
      "status": "Failed",
      "startTime": "2026-10-19T20:28:35.166007052Z",
      "endTime": "2026-10-19T20:47:38.665697527Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
            correlation ID: da09f0ee-1b1c-4dd0-986e-72ed6522e248
            client request ID: 773c74d7-3374-4a99-9619-c2776f1df3e0
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x13c01da76240>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x13c01e312060>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x13c01f5045e0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x13c01e3822a0>{
                        msg: "DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x13c01da76240>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x13c01e312060>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x13c01f5045e0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x13c01e3822a0>{
                        msg: "DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

time=2026-10-19T19:34:54.564Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T19:34:54.990Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T19:34:54.990Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T19:34:55.299Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-19T19:34:55.456Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-19T19:34:55.456Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T19:34:55.507Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T19:35:45.081Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T19:35:45.081Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x424386240f0>: 
    resource group "delayed-rbac-9xs9kgbdmnwv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-9xs9kgbdmnwv" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "name": "2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "status": "Failed",
      "startTime": "2026-10-19T19:53:00.471686176Z",
      "endTime": "2026-10-19T20:17:09.335861904Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
            correlation ID: e5bf541d-883b-4a45-98e4-199dc9b858a9
            client request ID: 39529ecb-f70a-4c21-a2de-abd0e5ac1e6b
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x424386240f0>: 
    resource group "delayed-rbac-9xs9kgbdmnwv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-9xs9kgbdmnwv" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "name": "2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "status": "Failed",
      "startTime": "2026-10-19T19:53:00.471686176Z",
      "endTime": "2026-10-19T20:17:09.335861904Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
            correlation ID: e5bf541d-883b-4a45-98e4-199dc9b858a9
            client request ID: 39529ecb-f70a-4c21-a2de-abd0e5ac1e6b
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x3d4749b3c078>: 
    resource group "customer-rg-j9j4x64vc846": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-j9j4x64vc846" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867",
      "name": "060bb367-1bd8-42f9-b696-549f7d9fb867",
      "status": "Failed",
      "startTime": "2026-10-19T20:02:54.979353355Z",
      "endTime": "2026-10-19T20:27:00.270376787Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ti2ch5ekujd757n62ntgfdpp1p6fpca still exists (deletion dispatched at 2026-10-19T20:02:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
            correlation ID: 1b181c6c-5659-46ac-a08f-39fecbba1c31
            client request ID: 190e259b-25d3-414f-bd78-f91c53d9c681
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x3d4749b3c078>: 
    resource group "customer-rg-j9j4x64vc846": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-j9j4x64vc846" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867",
      "name": "060bb367-1bd8-42f9-b696-549f7d9fb867",
      "status": "Failed",
      "startTime": "2026-10-19T20:02:54.979353355Z",
      "endTime": "2026-10-19T20:27:00.270376787Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ti2ch5ekujd757n62ntgfdpp1p6fpca still exists (deletion dispatched at 2026-10-19T20:02:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
            correlation ID: 1b181c6c-5659-46ac-a08f-39fecbba1c31
            client request ID: 190e259b-25d3-414f-bd78-f91c53d9c681
occurred

time=2026-10-19T19:36:46.694Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-10-19T19:36:47.222Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-19T19:36:47.222Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T19:36:47.440Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-10-19T19:36:47.554Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-10-19T19:36:47.554Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T19:36:47.600Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T19:42:29.171Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T19:42:29.171Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x28db69b98ae0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-f6gwn5qzckpq" to finish creating: GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "name": "55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "status": "Failed",
      "startTime": "2026-10-19T19:48:10.999588854Z",
      "endTime": "2026-10-19T20:07:15.605638501Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-wmvjh-tdsrw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-wmvjh-tdsrw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
            correlation ID: 43d991c8-f99d-4e5d-8930-6d417ef255cd
            client request ID: 24277a58-4912-4429-b89c-c6baeb4435fc
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x28db69b98ae0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-f6gwn5qzckpq" to finish creating: GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "name": "55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "status": "Failed",
      "startTime": "2026-10-19T19:48:10.999588854Z",
      "endTime": "2026-10-19T20:07:15.605638501Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-wmvjh-tdsrw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-wmvjh-tdsrw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
            correlation ID: 43d991c8-f99d-4e5d-8930-6d417ef255cd
            client request ID: 24277a58-4912-4429-b89c-c6baeb4435fc
occurred

alert fired 3 time(s) (1 unknown, 2 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T19:57:21Z
  Ended:       2026-10-19T20:23:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8677376-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti26r8lt041sjsk1mfsgims5nh4gjni-j2j6u5m5g1b2x8s", pod="azure-disk-csi-driver-controller-55fb85cb4b-zqvs9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti26r8lt041sjsk1mfsgims5nh4gjni-j2j6u5m5g1b2x8s/azure-disk-csi-driver-controller-55fb85cb4b-zqvs9 has been in a non-ready state for longer than 5 minutes.

time=2026-10-19T18:44:25.577Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-19T18:44:27.375Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-19T18:44:29.162Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=6167d0252600ec817c994890bab60d9173f655840adcabf122067da893127d1f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8937216-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6167d0252600ec817c994890bab60d9173f655840adcabf122067da893127d1f
time=2026-10-19T18:53:01.241Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Resources/deployments/6167d0252600ec817c994890bab60d9173f655840adcabf122067da893127d1f/operationStatuses/08584091714172679540\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Resources/deployments/profile-w5ahaumc2w6ei\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j8937216-svc/resourceAssociations/nsp-j8937216-svc-ra-d5a635c9-43bf-5de7-aaf2-5232b9ebe5d0\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j8937216-svc/resourceAssociations/nsp-j8937216-svc-ra-d5a635c9-43bf-5de7-aaf2-5232b9ebe5d0, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3895fbd87c80>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-89sq9p7645tk" to finish creating: GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7",
      "name": "111cac39-15a9-4a55-ab81-242565fc5fd7",
      "status": "Failed",
      "startTime": "2026-10-19T19:38:52.630267131Z",
      "endTime": "2026-10-19T19:57:56.018733569Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
            correlation ID: 439e72b2-9c87-4ffb-ac85-0abf45d7f0d2
            client request ID: 07db40a5-8bb3-4389-ac48-a0125b515b0e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3895fbd87c80>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-89sq9p7645tk" to finish creating: GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7",
      "name": "111cac39-15a9-4a55-ab81-242565fc5fd7",
      "status": "Failed",
      "startTime": "2026-10-19T19:38:52.630267131Z",
      "endTime": "2026-10-19T19:57:56.018733569Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
            correlation ID: 439e72b2-9c87-4ffb-ac85-0abf45d7f0d2
            client request ID: 07db40a5-8bb3-4389-ac48-a0125b515b0e
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:54:11Z
  Ended:       2026-10-19T19:08:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7279232-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti1cs38lrcs57mr7od4i6kk8vg0ki0m-u4p1g2o4u2r8x4h", pod="router-596877b669-ql8jm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti1cs38lrcs57mr7od4i6kk8vg0ki0m-u4p1g2o4u2r8x4h/router-596877b669-ql8jm has been in a non-ready state for longer than 5 minutes.

time=2026-10-19T18:14:28.628Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T18:14:29.083Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T18:14:29.083Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T18:14:29.257Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-19T18:14:29.460Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-19T18:14:29.460Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T18:14:29.517Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T18:15:22.178Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T18:15:22.178Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xdbc40b14b80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-9ndqj45cmqg2" to finish creating: GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "name": "a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "status": "Failed",
      "startTime": "2026-10-19T18:15:40.641153712Z",
      "endTime": "2026-10-19T18:44:47.329454995Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
            correlation ID: b9b321eb-17a4-4f82-91e4-993a13652950
            client request ID: f220b5f7-1c64-40ab-bc96-b0e5ef27428d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xdbc40b14b80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-9ndqj45cmqg2" to finish creating: GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "name": "a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "status": "Failed",
      "startTime": "2026-10-19T18:15:40.641153712Z",
      "endTime": "2026-10-19T18:44:47.329454995Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
            correlation ID: b9b321eb-17a4-4f82-91e4-993a13652950
            client request ID: f220b5f7-1c64-40ab-bc96-b0e5ef27428d
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: failed to create customer resources for second cluster "basic-hcp-cluster2"
Unexpected error:
    <*fmt.wrapError | 0x3cfd776cc680>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-vrgp2b" in resourcegroup="customer-rg-d7bzdbm7gf6t" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xooij5h335npw' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
            correlation ID: 0edba454-3af2-4d43-98da-9b281d7b5b9e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: failed to create customer resources for second cluster "basic-hcp-cluster2"
Unexpected error:
    <*fmt.wrapError | 0x3cfd776cc680>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-vrgp2b" in resourcegroup="customer-rg-d7bzdbm7gf6t" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xooij5h335npw' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
            correlation ID: 0edba454-3af2-4d43-98da-9b281d7b5b9e
            client request ID: unavailable
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:34:58Z
  Ended:       2026-10-19T19:02:56Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:35:57Z
  Ended:       2026-10-19T19:06:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:49:54Z
  Ended:       2026-10-19T19:02:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:52:56Z
  Ended:       2026-10-19T19:03:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:36:54Z
  Ended:       2026-10-19T19:06:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ab10566e-97b4-4b52-af7f-f5418105b4d1"
  Description: Pod ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:36:54Z
  Ended:       2026-10-19T19:06:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ab10566e-97b4-4b52-af7f-f5418105b4d1"
  Description: Pod ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 36 time(s) (11 unknown, 25 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:30:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-w2kfh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-w2kfh has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-7s4w7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-7s4w7 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-zqd9s", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-zqd9s has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-ls5fv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-ls5fv has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-nw4vv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-nw4vv has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-bhckh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-bhckh has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-2ssnn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-2ssnn has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-dgw65", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-dgw65 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-jjkgb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-jjkgb has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:30:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-m7jnr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-m7jnr has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-19T18:05:32Z
  Ended:       2026-10-19T18:30:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6568832-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster", pod="multus-admission-controller-6d89f9fc8b-dg594", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ti0ke1dfm7gh0tl4j86qi7e852u4p94-cilium-cluster/multus-admission-controller-6d89f9fc8b-dg594 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x39c1da55a940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-vfc45fcxtxqz" to finish creating: GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "name": "98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "status": "Failed",
      "startTime": "2026-10-19T18:03:22.205099398Z",
      "endTime": "2026-10-19T18:22:25.549095315Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
            correlation ID: 697bbb7b-4488-43cf-b80e-bef1ce56a7c0
            client request ID: 9c6f432c-bb1f-45d1-a1cc-acc4672dbd73
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x39c1da55a940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-vfc45fcxtxqz" to finish creating: GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "name": "98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "status": "Failed",
      "startTime": "2026-10-19T18:03:22.205099398Z",
      "endTime": "2026-10-19T18:22:25.549095315Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
            correlation ID: 697bbb7b-4488-43cf-b80e-bef1ce56a7c0
            client request ID: 9c6f432c-bb1f-45d1-a1cc-acc4672dbd73

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

time=2026-10-19T17:29:06.855Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T17:29:07.509Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T17:29:07.509Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T17:29:07.746Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-19T17:29:08.064Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-19T17:29:08.064Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T17:29:08.191Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T17:30:30.280Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T17:30:30.280Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

time=2026-10-19T16:50:11.764Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T16:50:12.180Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T16:50:12.180Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T16:50:12.348Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-19T16:50:12.613Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-19T16:50:12.613Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T16:50:12.683Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T16:51:02.039Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T16:51:02.040Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2d83e42499e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-85n46zxr29bc" to finish creating: GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da",
      "name": "eb476919-259d-4581-ba0c-41803ac281da",
      "status": "Failed",
      "startTime": "2026-10-19T17:15:51.295694688Z",
      "endTime": "2026-10-19T17:34:52.47411646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
            correlation ID: 2736a12e-0c94-445a-972d-f60f128898b0
            client request ID: 8ae6dbe5-3e36-4ee4-b803-b4d4643b03d4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2d83e42499e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-85n46zxr29bc" to finish creating: GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da",
      "name": "eb476919-259d-4581-ba0c-41803ac281da",
      "status": "Failed",
      "startTime": "2026-10-19T17:15:51.295694688Z",
      "endTime": "2026-10-19T17:34:52.47411646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
            correlation ID: 2736a12e-0c94-445a-972d-f60f128898b0
            client request ID: 8ae6dbe5-3e36-4ee4-b803-b4d4643b03d4
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-bbzfqrxldsr2/cluster-candidate-5-1-vz4k7j should provision
Unexpected error:
    <*fmt.wrapError | 0x399878fce5c0>: 
    failed to create HCP cluster cluster-candidate-5-1-vz4k7j: failed waiting for cluster="cluster-candidate-5-1-vz4k7j" in resourcegroup="rg-candidate-5-1-bbzfqrxldsr2" to finish creating: GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "name": "f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "status": "Failed",
      "startTime": "2026-10-19T16:52:45.887911577Z",
      "endTime": "2026-10-19T17:21:46.629500013Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
            correlation ID: 4cf69c21-e6dd-4c6d-8bd7-3644b7285946
            client request ID: 2f22a1ad-d176-4f31-9493-27c1ef58e861
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-bbzfqrxldsr2/cluster-candidate-5-1-vz4k7j should provision
Unexpected error:
    <*fmt.wrapError | 0x399878fce5c0>: 
    failed to create HCP cluster cluster-candidate-5-1-vz4k7j: failed waiting for cluster="cluster-candidate-5-1-vz4k7j" in resourcegroup="rg-candidate-5-1-bbzfqrxldsr2" to finish creating: GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "name": "f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "status": "Failed",
      "startTime": "2026-10-19T16:52:45.887911577Z",
      "endTime": "2026-10-19T17:21:46.629500013Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
            correlation ID: 4cf69c21-e6dd-4c6d-8bd7-3644b7285946
            client request ID: 2f22a1ad-d176-4f31-9493-27c1ef58e861
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T17:14:26Z
  Ended:       2026-10-19T17:38:23Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8l75h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T17:26:26Z
  Ended:       2026-10-19T17:38:23Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8l75h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T17:17:20Z
  Ended:       2026-10-19T17:42:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="001c2a8a-fc34-4e11-96e9-d8a9bf0c0d4a"
  Description: Pod ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T17:17:20Z
  Ended:       2026-10-19T17:42:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="001c2a8a-fc34-4e11-96e9-d8a9bf0c0d4a"
  Description: Pod ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

time=2026-10-19T16:02:01.049Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T16:02:01.490Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T16:02:01.490Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T16:02:01.775Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-19T16:02:01.979Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-19T16:02:01.979Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T16:02:02.044Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T16:02:57.648Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T16:02:57.648Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x2ba9acdcc318>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-pp8lljnf5t7j" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "name": "025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "status": "Failed",
      "startTime": "2026-10-19T16:45:45.996504863Z",
      "endTime": "2026-10-19T17:04:48.768522414Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-7v7tf-6wgpp: Failed: availabilitysets failed to create or update. err: failed to get existing resource cni-cilium-pp8lljnf5t7j--managed/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-pp8lljnf5t7j--managed/providers/Microsoft.Compute/availabilitySets/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after ad... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
            correlation ID: c38c42ba-d929-4393-9a38-4d460e6cc1c1
            client request ID: 991ecdc6-7a24-4e68-a023-12a15de7fc26
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x2ba9acdcc318>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-pp8lljnf5t7j" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "name": "025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "status": "Failed",
      "startTime": "2026-10-19T16:45:45.996504863Z",
      "endTime": "2026-10-19T17:04:48.768522414Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-7v7tf-6wgpp: Failed: availabilitysets failed to create or update. err: failed to get existing resource cni-cilium-pp8lljnf5t7j--managed/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-pp8lljnf5t7j--managed/providers/Microsoft.Compute/availabilitySets/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after ad... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
            correlation ID: c38c42ba-d929-4393-9a38-4d460e6cc1c1
            client request ID: 991ecdc6-7a24-4e68-a023-12a15de7fc26
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1051bd7af500>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-p7tdc2" in resourcegroup="rg-np-version-upgrade-p7tdc2-lnnmsq2r4w9h" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "name": "70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "status": "Failed",
      "startTime": "2026-10-19T17:02:00.685391332Z",
      "endTime": "2026-10-19T17:21:08.269829146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
            correlation ID: fb9e2f48-428d-4f30-8c61-d916aa767023
            client request ID: 93ac57ce-9b86-4ca4-8a14-339c496ee50f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1051bd7af500>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-p7tdc2" in resourcegroup="rg-np-version-upgrade-p7tdc2-lnnmsq2r4w9h" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "name": "70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "status": "Failed",
      "startTime": "2026-10-19T17:02:00.685391332Z",
      "endTime": "2026-10-19T17:21:08.269829146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
            correlation ID: fb9e2f48-428d-4f30-8c61-d916aa767023
            client request ID: 93ac57ce-9b86-4ca4-8a14-339c496ee50f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 46 time(s) (22 unknown, 24 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T16:40:34Z
  Ended:       2026-10-19T16:56:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfqmn4vb4jp0h2nb17kirlvku8jif-r3p4f2r1s4c7j7k", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfqmn4vb4jp0h2nb17kirlvku8jif-r3p4f2r1s4c7j7k/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T16:41:37Z
  Ended:       2026-10-19T16:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfuaumdfma93r70770i2lk0ceibuu-l1u4a6z7l3x3m9t", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfuaumdfma93r70770i2lk0ceibuu-l1u4a6z7l3x3m9t/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T16:41:38Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfscjl15blbh8mcver1v3o3u4u0ao-y6i4i5p1q7w7n4v", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfscjl15blbh8mcver1v3o3u4u0ao-y6i4i5p1q7w7n4v/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T16:41:38Z
  Ended:       2026-10-19T16:57:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfqlloamhkfi0hhbkskh026c53tqv-x1z4l6s3s3f6f6o", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfqlloamhkfi0hhbkskh026c53tqv-x1z4l6s3s3f6f6o/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T16:42:35Z
  Ended:       2026-10-19T16:56:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfu9i3hibd7t67f8k4ikin22uvov0-ksm-hcp-metrics", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfu9i3hibd7t67f8k4ikin22uvov0-ksm-hcp-metrics/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T16:42:35Z
  Ended:       2026-10-19T16:56:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfsq2gk0ogv6igthdon43ohkqcll6-o3h6i5c2r6h8g6m", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfsq2gk0ogv6igthdon43ohkqcll6-o3h6i5c2r6h8g6m/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T16:42:35Z
  Ended:       2026-10-19T16:56:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfs4dgfss7ak7ob076ptv9s5b15c3-l9k0d8k9y9c4d6s", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfs4dgfss7ak7ob076ptv9s5b15c3-l9k0d8k9y9c4d6s/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T16:42:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfrp642h257s3p2mm1u1cbq24avuc-p0w0i8v2l6f3j9u", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfrp642h257s3p2mm1u1cbq24avuc-p0w0i8v2l6f3j9u/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T16:42:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfq8h15dkpqo3c05ofeksdhql6h0u-z7z1l9c1f6v5r1j", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfq8h15dkpqo3c05ofeksdhql6h0u-z7z1l9c1f6v5r1j/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T16:42:36Z
  Ended:       2026-10-19T16:56:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfstjgcqtr9i5gin0ic7r9c1odfdl-n3k2a6t9l8q5x7n", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfstjgcqtr9i5gin0ic7r9c1odfdl-n3k2a6t9l8q5x7n/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-19T16:42:36Z
  Ended:       2026-10-19T16:56:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfqm9keb8d2iec07muheaef524p1o-w4e1x7i1r6r0j5e", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfqm9keb8d2iec07muheaef524p1o-w4e1x7i1r6r0j5e/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-19T16:42:36Z
  Ended:       2026-10-19T16:56:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfrbnige07o90mmp5ao4p4ebo39p9-g0c7o0t4o8h3w0s", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfrbnige07o90mmp5ao4p4ebo39p9-g0c7o0t4o8h3w0s/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-19T16:42:36Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfricr8n4cfb8phakp84govl5sbp4-ea-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfricr8n4cfb8phakp84govl5sbp4-ea-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-19T16:43:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfs8va9j8splokkrcarauj3mssv2q-l5u5w9u0a1b4y4g", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfs8va9j8splokkrcarauj3mssv2q-l5u5w9u0a1b4y4g/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-19T16:43:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfrnukiqqd5c22t25k5k1vr8fpt73-j4c2r5k0m9r0j9c", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfrnukiqqd5c22t25k5k1vr8fpt73-j4c2r5k0m9r0j9c/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-19T16:43:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfspu00q5i9vof2vd94tgeok1kr3n-basic-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfspu00q5i9vof2vd94tgeok1kr3n-basic-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-19T16:43:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvftigig9mm991ouavfjlt16738njh-v0a4e1f5j5b2m0a", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvftigig9mm991ouavfjlt16738njh-v0a4e1f5j5b2m0a/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-19T16:44:35Z
  Ended:       2026-10-19T16:56:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfqju2ouj4onjdmo5mfj9dqmukjro-v20261001", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfqju2ouj4onjdmo5mfj9dqmukjro-v20261001/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-19T16:44:35Z
  Ended:       2026-10-19T16:56:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvgh1ga2vr8rfio4pgkn2vv1697h7e-cilium-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvgh1ga2vr8rfio4pgkn2vv1697h7e-cilium-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-19T16:44:35Z
  Ended:       2026-10-19T16:57:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfrkvqcg5e1rt0aettqt8df1df3pl-b4t6o0h2z3h3r6l", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfrkvqcg5e1rt0aettqt8df1df3pl-b4t6o0h2z3h3r6l/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-19T16:44:35Z
  Ended:       2026-10-19T16:58:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvfqh42qu5vrcpacs896nrie1l1pul-z1u5u6p0i7q4q4f", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvfqh42qu5vrcpacs896nrie1l1pul-z1u5u6p0i7q4q4f/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-19T16:45:35Z
  Ended:       2026-10-19T16:56:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thvgrpsbbr2rlo3i4jfsgsa1mbns4b8-agg-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thvgrpsbbr2rlo3i4jfsgsa1mbns4b8-agg-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T16:51:41Z
  Ended:       2026-10-19T16:57:41Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2325504/providers/microsoft.monitor/accounts/hcps-j2325504", namespace="ocm-arohcpci01-2thvfscjl15blbh8mcver1v3o3u4u0ao-y6i4i5p1q7w7n4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2wfh7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvfscjl15blbh8mcver1v3o3u4u0ao-y6i4i5p1q7w7n4v/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T16:51:41Z
  Ended:       2026-10-19T16:57:41Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2325504/providers/microsoft.monitor/accounts/hcps-j2325504", namespace="ocm-arohcpci01-2thvfqlloamhkfi0hhbkskh026c53tqv-x1z4l6s3s3f6f6o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2wfh7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvfqlloamhkfi0hhbkskh026c53tqv-x1z4l6s3s3f6f6o/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T16:51:41Z
  Ended:       2026-10-19T16:57:42Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2325504/providers/microsoft.monitor/accounts/hcps-j2325504", namespace="ocm-arohcpci01-2thvfscjl15blbh8mcver1v3o3u4u0ao-y6i4i5p1q7w7n4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2wfh7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvfscjl15blbh8mcver1v3o3u4u0ao-y6i4i5p1q7w7n4v/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T16:51:41Z
  Ended:       2026-10-19T16:57:41Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2325504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2325504/providers/microsoft.monitor/accounts/hcps-j2325504", namespace="ocm-arohcpci01-2thvfqlloamhkfi0hhbkskh026c53tqv-x1z4l6s3s3f6f6o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2wfh7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvfqlloamhkfi0hhbkskh026c53tqv-x1z4l6s3s3f6f6o/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3f88271a0ea0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2lvnhrzd5g7t" to finish creating: GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "name": "9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "status": "Failed",
      "startTime": "2026-10-19T16:38:57.011707926Z",
      "endTime": "2026-10-19T16:58:00.847122787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: NodeDoesNotExist: Node does not exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: Deleting: virtualmachine deleting\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
            correlation ID: 78955e10-7740-4b87-a377-073a6ecddbd0
            client request ID: a8ccc39c-63f9-475d-8c01-623fcd9a0f40
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3f88271a0ea0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2lvnhrzd5g7t" to finish creating: GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "name": "9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "status": "Failed",
      "startTime": "2026-10-19T16:38:57.011707926Z",
      "endTime": "2026-10-19T16:58:00.847122787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: NodeDoesNotExist: Node does not exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: Deleting: virtualmachine deleting\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
            correlation ID: 78955e10-7740-4b87-a377-073a6ecddbd0
            client request ID: a8ccc39c-63f9-475d-8c01-623fcd9a0f40
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T16:38:15Z
  Ended:       2026-10-19T16:53:16Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5762560-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T16:27:31Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8016256-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thv7hcqcal2f2hpopre8d6de2pirqto-v7m5p2q2k9d1x6l", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2thv7hcqcal2f2hpopre8d6de2pirqto-v7m5p2q2k9d1x6l/etcd-1 has been in a non-ready state for longer than 5 minutes.

time=2026-10-19T15:56:31.916Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-10-19T15:56:32.339Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-19T15:56:32.339Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T15:56:32.526Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-10-19T15:56:32.659Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-10-19T15:56:32.659Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T15:56:32.731Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T16:02:20.224Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T16:02:20.224Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2d6e7c8ac020>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-t27g7scwdvdr" to finish creating: GET https://rp.j9195008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04516756-3c0d-4cb8-9bea-3b83299b21af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04516756-3c0d-4cb8-9bea-3b83299b21af",
      "name": "04516756-3c0d-4cb8-9bea-3b83299b21af",
      "status": "Failed",
      "startTime": "2026-10-19T16:30:35.481364162Z",
      "endTime": "2026-10-19T16:49:44.492413303Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-t27g7scwdvdr--managed/providers/Microsoft.Compute/virtualMachines/j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-t27g7scwdvdr--managed/providers/Microsoft.Compute/virtualMachines/j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9195008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04516756-3c0d-4cb8-9bea-3b83299b21af
            correlation ID: 164cf44e-1a4c-43ef-ad03-502d13c22da5
            client request ID: 949d080f-05a2-40b5-b875-b9fbced5c91f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2d6e7c8ac020>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-t27g7scwdvdr" to finish creating: GET https://rp.j9195008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04516756-3c0d-4cb8-9bea-3b83299b21af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04516756-3c0d-4cb8-9bea-3b83299b21af",
      "name": "04516756-3c0d-4cb8-9bea-3b83299b21af",
      "status": "Failed",
      "startTime": "2026-10-19T16:30:35.481364162Z",
      "endTime": "2026-10-19T16:49:44.492413303Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-t27g7scwdvdr--managed/providers/Microsoft.Compute/virtualMachines/j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-t27g7scwdvdr--managed/providers/Microsoft.Compute/virtualMachines/j9y3e8x7n5w0j9t-autoscale-noaz-wr9r9-5zvr8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9195008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04516756-3c0d-4cb8-9bea-3b83299b21af
            correlation ID: 164cf44e-1a4c-43ef-ad03-502d13c22da5
            client request ID: 949d080f-05a2-40b5-b875-b9fbced5c91f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-sjxzmq"
Unexpected error:
    <*fmt.wrapError | 0x3e0c69717cc0>: 
    DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3e0c6a7f4740>{
            msg: "DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-sjxzmq"
Unexpected error:
    <*fmt.wrapError | 0x3e0c69717cc0>: 
    DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3e0c6a7f4740>{
            msg: "DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0x1ffd80fcc600>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1ffd80c0ac80>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1ffd811f6800>{
                    msg: "DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1ffd80d0e5e0>{
                        msg: "DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0x1ffd80fcc600>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1ffd80c0ac80>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1ffd811f6800>{
                    msg: "DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1ffd80d0e5e0>{
                        msg: "DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 92 time(s) (91 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-52xls", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-52xls has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-8pj5j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-8pj5j has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-872zw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-872zw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-hlzns", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-hlzns has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-cz2v8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-cz2v8 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-mw5bq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-mw5bq has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-mlxzc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-mlxzc has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-zldjh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-zldjh has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-75d6b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-75d6b has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-lzmgv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-lzmgv has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-b88nc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-b88nc has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-dzd9v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-dzd9v has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-8jtmm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-8jtmm has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-kwh7b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-kwh7b has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-9b4lv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-9b4lv has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-4b762", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-4b762 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-ptzhd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-ptzhd has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-ntc8c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-ntc8c has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-c274j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-c274j has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-c4w8j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-c4w8j has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-ctjgl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-ctjgl has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-r7c52", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-r7c52 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-dpqxk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-dpqxk has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-c4lpf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-c4lpf has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-k9pxx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-k9pxx has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-79g9t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-79g9t has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-nqjtj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-nqjtj has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-w5zkh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-w5zkh has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-st8sn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-st8sn has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-p56lh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-p56lh has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-clhxc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-clhxc has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-6cgkc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-6cgkc has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-4qmk4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-4qmk4 has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-psqcc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-psqcc has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-kpddb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-kpddb has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-5pkwb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-5pkwb has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-7x9vn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-7x9vn has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-bhl9l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-bhl9l has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-hmx5b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-hmx5b has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-kwrch", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-kwrch has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-whdp8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-whdp8 has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-zmr7m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-zmr7m has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-lhmw6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-lhmw6 has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-p5b9c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-p5b9c has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-zfnjh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-zfnjh has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-shf54", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-shf54 has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-k9fm5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-k9fm5 has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-tcl54", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-tcl54 has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-z84cl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-z84cl has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-k72n4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-k72n4 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-9n9rh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-9n9rh has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-zbjdg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-zbjdg has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-jdkf6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-jdkf6 has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-rtqkz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-rtqkz has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-w4sj4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-w4sj4 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-7tjw6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-7tjw6 has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-stzcg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-stzcg has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-tgdzr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-tgdzr has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-svk9f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-svk9f has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-sv2t5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-sv2t5 has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-rpf9p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-rpf9p has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-qznl4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-qznl4 has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:37:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-45tq6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-45tq6 has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-h55sn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-h55sn has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-twzxj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-twzxj has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-kz995", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-kz995 has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-fv8xv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-fv8xv has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-bbz4w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-bbz4w has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-c6ss9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-c6ss9 has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-6vj24", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-6vj24 has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-qbjp9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-qbjp9 has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-7dn78", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-7dn78 has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-txj72", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-txj72 has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-mwfqs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-mwfqs has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-5bt4t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-5bt4t has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-s92xf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-s92xf has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-f5nxl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-f5nxl has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-pwdbs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-pwdbs has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:38:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-6nt9d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-6nt9d has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-64c6c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-64c6c has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-lh56t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-lh56t has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-kqdst", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-kqdst has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-w824m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-w824m has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-19T16:15:27Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-d6q4r", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-d6q4r has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-19T16:15:28Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-n59nc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-n59nc has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-19T16:15:28Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-lzjfr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-lzjfr has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-19T16:15:28Z
  Ended:       2026-10-19T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-blh9z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-blh9z has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-19T16:15:28Z
  Ended:       2026-10-19T16:33:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-8lcct", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-8lcct has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-19T16:15:28Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-tgq4m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-tgq4m has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-19T16:15:28Z
  Ended:       2026-10-19T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ignition-server-proxy-5678699d94-6s756", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ignition-server-proxy-5678699d94-6s756 has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-19T16:16:31Z
  Ended:       2026-10-19T16:38:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5037184-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h", pod="ovnkube-control-plane-566d46bb86-j5r7h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thuupsd8j59p35gb9lk4m4qnu0aiu8m-i5w5x8k4i5q6a4h/ovnkube-control-plane-566d46bb86-j5r7h has been in a non-ready state for longer than 5 minutes.

time=2026-10-19T15:30:44.609Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-19T15:30:46.080Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-19T15:30:46.080Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-19T15:30:46.888Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-19T15:30:46.889Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-19T15:30:46.889Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j8401536-svc
time=2026-10-19T15:45:50.026Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"

time=2026-10-19T15:27:02.707Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T15:27:03.206Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T15:27:03.206Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T15:27:03.631Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-19T15:27:03.853Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-19T15:27:03.853Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T15:27:03.915Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T15:28:05.016Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T15:28:05.016Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x3b24ac0020c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-p69pvshqvq9p" to finish creating: GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "name": "c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "status": "Failed",
      "startTime": "2026-10-19T15:56:13.653725725Z",
      "endTime": "2026-10-19T16:15:18.87162238Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
            correlation ID: 5cc9d41d-adc2-4131-802d-17d3ab484012
            client request ID: b9e380f6-f1bc-4da7-bbfb-dbb4f50cb471
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x3b24ac0020c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-p69pvshqvq9p" to finish creating: GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "name": "c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "status": "Failed",
      "startTime": "2026-10-19T15:56:13.653725725Z",
      "endTime": "2026-10-19T16:15:18.87162238Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
            correlation ID: 5cc9d41d-adc2-4131-802d-17d3ab484012
            client request ID: b9e380f6-f1bc-4da7-bbfb-dbb4f50cb471
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 8 time(s) (2 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T15:26:26Z
  Ended:       2026-10-19T15:47:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8150784-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thu6ecd4n5sg6hvpveee3vhkda08haq-j0e0i2a0u5b4n0x", pod="multus-admission-controller-5d6cb76b4d-pzd4m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thu6ecd4n5sg6hvpveee3vhkda08haq-j0e0i2a0u5b4n0x/multus-admission-controller-5d6cb76b4d-pzd4m has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T15:26:26Z
  Ended:       2026-10-19T15:43:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8150784-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thu6ecd4n5sg6hvpveee3vhkda08haq-j0e0i2a0u5b4n0x", pod="multus-admission-controller-5d6cb76b4d-qpjgm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thu6ecd4n5sg6hvpveee3vhkda08haq-j0e0i2a0u5b4n0x/multus-admission-controller-5d6cb76b4d-qpjgm has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T15:19:57Z
  Ended:       2026-10-19T15:33:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4950784-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T15:24:05Z
  Ended:       2026-10-19T15:33:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4950784-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

time=2026-10-19T14:43:12.749Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-19T14:43:13.235Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1
time=2026-10-19T14:43:13.684Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 deployment=299b7403a208c1c427ae8ebbcd18bb4f4fb4ff513a91ec66ec3b8e76bf4affb0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1534464-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F299b7403a208c1c427ae8ebbcd18bb4f4fb4ff513a91ec66ec3b8e76bf4affb0
time=2026-10-19T14:43:43.960Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Resources/deployments/299b7403a208c1c427ae8ebbcd18bb4f4fb4ff513a91ec66ec3b8e76bf4affb0/operationStatuses/08584091858920261215\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Resources/deployments/profile-xcn2acurowiq2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j1534464-mgmt-1/resourceAssociations/nsp-j1534464-mgmt-1-ra-66bb4858-d2d4-5bd8-8103-db26bfc7bd94\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j1534464-mgmt-1/resourceAssociations/nsp-j1534464-mgmt-1-ra-66bb4858-d2d4-5bd8-8103-db26bfc7bd94, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

alert fired 37 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-vft2k", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-vft2k has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-mxpsb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-mxpsb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-x658v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-x658v has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-vf6t4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-vf6t4 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-n7j7q", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-n7j7q has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-27md8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-27md8 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-284f9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-284f9 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-xk84r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-xk84r has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T15:06:10Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-8s8vm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-8s8vm has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-9zjnm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-9zjnm has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-xkdrg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-xkdrg has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-fdrzp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-fdrzp has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-9gsrl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-9gsrl has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-486j8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-486j8 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-22pvb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-22pvb has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-mm4rn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-mm4rn has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-8nbrp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-8nbrp has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-2fb6m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-2fb6m has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-8bk65", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-8bk65 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-vxx9r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-vxx9r has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-2vsrw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-2vsrw has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-bzjbl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-bzjbl has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-q24x7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-q24x7 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-b5lns", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-b5lns has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-4v9ts", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-4v9ts has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-njzt7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-njzt7 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-n5456", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-n5456 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-49x29", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-49x29 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-5vj7r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-5vj7r has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-6ljqn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-6ljqn has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-7fcjd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-7fcjd has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-47qtx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-47qtx has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-59j4f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-59j4f has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-4gxhr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-4gxhr has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-kpjg2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-kpjg2 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-cqht6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-cqht6 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-19T15:06:11Z
  Ended:       2026-10-19T15:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0466176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m", pod="kube-controller-manager-85d56cd499-ff847", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thtv10b2qlgq1ue1q40ce21ebfh5jeu-y9e3r7k6y9l6j6m/kube-controller-manager-85d56cd499-ff847 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x3eb77839e738>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x3eb77839e738>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:23:28Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-svc", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:29Z
  Ended:       2026-10-19T15:04:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:47:27Z
  Ended:       2026-10-19T15:07:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:23:34Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-svc", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:35Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:35Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:23:41Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-svc", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:41Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:41Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:14Z
  Ended:       2026-10-19T15:04:13Z
  Severity:    Sev2
  Labels:      alertname="KubeletDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: Kubelet has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:47:17Z
  Ended:       2026-10-19T15:07:16Z
  Severity:    Sev2
  Labels:      alertname="KubeletDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: Kubelet has disappeared from Prometheus target discovery.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:54Z
  Ended:       2026-10-19T15:06:57Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:46:54Z
  Ended:       2026-10-19T15:04:55Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:52Z
  Ended:       2026-10-19T15:04:51Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:46:52Z
  Ended:       2026-10-19T15:06:52Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x62872c04378>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x62872c04378>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1a6b532d3880>: 
    DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1a6b532d22c0>{
            msg: "DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1a6b532d3880>: 
    DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1a6b532d22c0>{
            msg: "DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x25effd432cd8>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x25effd432cd8>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x2e8edc927b0>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x2e8edc927b0>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x106d601dad60>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-dgl2pkrhlxx4" to finish creating: GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "name": "87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "status": "Failed",
      "startTime": "2026-10-19T14:24:33.557951156Z",
      "endTime": "2026-10-19T14:43:40.561544029Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine des-encrypt-des-np-rhpbh-rrqrt: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine des-encrypt-des-np-rhpbh-rrqrt: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
            correlation ID: 504c59a4-77a5-4f21-a6c2-acde02224bf6
            client request ID: fb1b6d77-0e4c-4919-b3ea-36fd2e07f63c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x106d601dad60>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-dgl2pkrhlxx4" to finish creating: GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "name": "87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "status": "Failed",
      "startTime": "2026-10-19T14:24:33.557951156Z",
      "endTime": "2026-10-19T14:43:40.561544029Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine des-encrypt-des-np-rhpbh-rrqrt: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine des-encrypt-des-np-rhpbh-rrqrt: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
            correlation ID: 504c59a4-77a5-4f21-a6c2-acde02224bf6
            client request ID: fb1b6d77-0e4c-4919-b3ea-36fd2e07f63c
occurred

alert fired 12 time(s) (6 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:16:02Z
  Ended:       2026-10-19T14:42:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3999104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o", pod="cloud-network-config-controller-6bd4b474cb-vrlfj", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o/cloud-network-config-controller-6bd4b474cb-vrlfj has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:16:02Z
  Ended:       2026-10-19T14:42:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3999104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o", pod="cloud-network-config-controller-6bd4b474cb-2kncv", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o/cloud-network-config-controller-6bd4b474cb-2kncv has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T14:16:03Z
  Ended:       2026-10-19T14:42:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3999104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o", pod="cloud-network-config-controller-6bd4b474cb-fbnwx", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o/cloud-network-config-controller-6bd4b474cb-fbnwx has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T14:16:03Z
  Ended:       2026-10-19T14:42:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3999104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o", pod="cloud-network-config-controller-6bd4b474cb-bs5m5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o/cloud-network-config-controller-6bd4b474cb-bs5m5 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T14:16:03Z
  Ended:       2026-10-19T14:42:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3999104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tht84d4qs8a785c9tlhvqhejsm8ekg7-v6o7e3k6y0b2o1n", pod="cloud-network-config-controller-6db797f7fb-zccvt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tht84d4qs8a785c9tlhvqhejsm8ekg7-v6o7e3k6y0b2o1n/cloud-network-config-controller-6db797f7fb-zccvt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T14:16:03Z
  Ended:       2026-10-19T14:42:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3999104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o", pod="cloud-network-config-controller-6bd4b474cb-dqphz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tht8gtrach45bq9eke6oba1q2fss001-w5j4t6r0g6q1i4o/cloud-network-config-controller-6bd4b474cb-dqphz has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:15:48Z
  Ended:       2026-10-19T14:29:51Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3999104-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x2013d3ee0768>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x2013d3ee0768>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xacc0f2d2440>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-jlnfb8x8vq5b" to finish creating: GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8",
      "name": "d0f38151-9c16-4736-8305-cfdc47700fc8",
      "status": "Failed",
      "startTime": "2026-10-19T13:29:55.150212587Z",
      "endTime": "2026-10-19T13:48:55.883771022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-fgcc4 to exist\nMachine ephemeral-disk-ephemeral-np-pscp5-w4psz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-w4psz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: Failed: interfaces failed to create or update. err: failed to get existing resource ephemeral-osdisk-jlnfb8x8vq5b--managed/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Network/networkInterfaces/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '10' seconds after additional token... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
            correlation ID: 346dd7ce-e606-492f-925a-aa985115ed46
            client request ID: 9eb4247b-08f9-4cb3-a183-7df951ef2271
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xacc0f2d2440>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-jlnfb8x8vq5b" to finish creating: GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8",
      "name": "d0f38151-9c16-4736-8305-cfdc47700fc8",
      "status": "Failed",
      "startTime": "2026-10-19T13:29:55.150212587Z",
      "endTime": "2026-10-19T13:48:55.883771022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-fgcc4 to exist\nMachine ephemeral-disk-ephemeral-np-pscp5-w4psz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-w4psz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: Failed: interfaces failed to create or update. err: failed to get existing resource ephemeral-osdisk-jlnfb8x8vq5b--managed/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Network/networkInterfaces/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '10' seconds after additional token... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
            correlation ID: 346dd7ce-e606-492f-925a-aa985115ed46
            client request ID: 9eb4247b-08f9-4cb3-a183-7df951ef2271
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4ksqkw" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x875c0f26cc0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4ksqkw: failed waiting for cluster="cp-ystream-upgrade-4-21-4ksqkw" in resourcegroup="rg-cp-ystream-upgrade-4-21-smptp4v58hz8" to finish creating: GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed4c5b14-08c2-420a-930b-887ecdc0b2b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed4c5b14-08c2-420a-930b-887ecdc0b2b3",
      "name": "ed4c5b14-08c2-420a-930b-887ecdc0b2b3",
      "status": "Failed",
      "startTime": "2026-10-19T13:20:10.282967129Z",
      "endTime": "2026-10-19T13:49:15.52369038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed4c5b14-08c2-420a-930b-887ecdc0b2b3
            correlation ID: 885d6f5f-5f9d-4eb3-8a13-c8dddc468bfa
            client request ID: 1533b694-7687-4dd7-bde1-4f961ff6ec29
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4ksqkw" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x875c0f26cc0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4ksqkw: failed waiting for cluster="cp-ystream-upgrade-4-21-4ksqkw" in resourcegroup="rg-cp-ystream-upgrade-4-21-smptp4v58hz8" to finish creating: GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed4c5b14-08c2-420a-930b-887ecdc0b2b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed4c5b14-08c2-420a-930b-887ecdc0b2b3",
      "name": "ed4c5b14-08c2-420a-930b-887ecdc0b2b3",
      "status": "Failed",
      "startTime": "2026-10-19T13:20:10.282967129Z",
      "endTime": "2026-10-19T13:49:15.52369038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed4c5b14-08c2-420a-930b-887ecdc0b2b3
            correlation ID: 885d6f5f-5f9d-4eb3-8a13-c8dddc468bfa
            client request ID: 1533b694-7687-4dd7-bde1-4f961ff6ec29
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x2d77cef36660>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x2d77cef36660>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T13:49:48Z
  Ended:       2026-10-19T13:59:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3eeb43ae-342b-4c43-8ac2-1b770f3075a4"
  Description: Pod ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T13:50:49Z
  Ended:       2026-10-19T13:59:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3eeb43ae-342b-4c43-8ac2-1b770f3075a4"
  Description: Pod ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T13:40:54Z
  Ended:       2026-10-19T14:06:55Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vsdh5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T13:40:54Z
  Ended:       2026-10-19T14:06:55Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vsdh5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T13:54:50Z
  Ended:       2026-10-19T14:07:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vsdh5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T13:55:53Z
  Ended:       2026-10-19T14:07:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vsdh5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x1162a0339230>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x1162a0339230>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x34e7945e0180>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x34e7945e0180>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x41704badb18>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x41704badb18>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:131]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0x1dd2fd780100>: 
    failed to get graph client: identify caller: get token: ClientSecretCredential: server response error:
     context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
    {
        msg: "failed to get graph client: identify caller: get token: ClientSecretCredential: server response error:\n context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret",
        err: <*fmt.wrapError | 0x1dd2fd7800e0>{
            msg: "identify caller: get token: ClientSecretCredential: server response error:\n context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret",
            err: <*fmt.wrapError | 0x1dd2fd7800c0>{
                msg: "get token: ClientSecretCredential: server response error:\n context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret",
                err: <*azidentity.AuthenticationFailedError | 0x1dd2fee1c150>{
                    RawResponse: nil,
                    credType: "ClientSecretCredential",
                    message: "server response error:\n context deadline exceeded",
                    omitResponse: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:131]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0x1dd2fd780100>: 
    failed to get graph client: identify caller: get token: ClientSecretCredential: server response error:
     context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
    {
        msg: "failed to get graph client: identify caller: get token: ClientSecretCredential: server response error:\n context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret",
        err: <*fmt.wrapError | 0x1dd2fd7800e0>{
            msg: "identify caller: get token: ClientSecretCredential: server response error:\n context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret",
            err: <*fmt.wrapError | 0x1dd2fd7800c0>{
                msg: "get token: ClientSecretCredential: server response error:\n context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret",
                err: <*azidentity.AuthenticationFailedError | 0x1dd2fee1c150>{
                    RawResponse: nil,
                    credType: "ClientSecretCredential",
                    message: "server response error:\n context deadline exceeded",
                    omitResponse: false,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-2qp7lk"
Unexpected error:
    <*fmt.wrapError | 0xd92a3e87160>: 
    DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xd92a3f627c0>{
            msg: "DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-2qp7lk"
Unexpected error:
    <*fmt.wrapError | 0xd92a3e87160>: 
    DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xd92a3f627c0>{
            msg: "DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xd36763f9420>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-q6wh6zbcv2x2" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997",
      "name": "42423002-b716-49bf-b44c-2e1fd4733997",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.508047813Z",
      "endTime": "2026-10-19T11:00:34.28001697Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
            correlation ID: 1aed53cb-56a1-406f-8bbf-9de9f16e4e1c
            client request ID: 90f415c0-66a9-4a24-b9de-2db2faf18c9c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xd36763f9420>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-q6wh6zbcv2x2" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997",
      "name": "42423002-b716-49bf-b44c-2e1fd4733997",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.508047813Z",
      "endTime": "2026-10-19T11:00:34.28001697Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
            correlation ID: 1aed53cb-56a1-406f-8bbf-9de9f16e4e1c
            client request ID: 90f415c0-66a9-4a24-b9de-2db2faf18c9c
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x1c5cb893a690>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-h758j8sh8qhg" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-h758j8sh8qhg, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-h758j8sh8qhg\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-h758j8sh8qhg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1c5cb889c060>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-h758j8sh8qhg",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x1c5cb893a690>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-h758j8sh8qhg" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-h758j8sh8qhg, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-h758j8sh8qhg\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-h758j8sh8qhg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1c5cb889c060>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-h758j8sh8qhg",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x169a1fa14360>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-pvwnm5h2lwhh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "name": "201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:39.240419381Z",
      "endTime": "2026-10-19T11:00:44.480106321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, cluster-node-tuning-operator, redhat-marketplace-catalog, catalog-operator, cluster-network-operator, certified-operators-catalog, olm-collect-profiles, openshift-controller-manager, packageserver, machine-approver, cluster-storage-operator, cluster-version-operator, cluster-policy-controller, openshift-route-controller-manager, community-operators-catalog, cluster-image-registry-operator, dns-operator, konnectivity-agent, olm-operator, azure-cloud-controller-manager, csi-snapshot-controller-operator, hosted-cluster-config-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
            correlation ID: a21d9aa7-da4e-4f79-9da1-353faa5bb8d0
            client request ID: 457b6d19-40ff-4036-a44f-729e35535125
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x169a1fa14360>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-pvwnm5h2lwhh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "name": "201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:39.240419381Z",
      "endTime": "2026-10-19T11:00:44.480106321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, cluster-node-tuning-operator, redhat-marketplace-catalog, catalog-operator, cluster-network-operator, certified-operators-catalog, olm-collect-profiles, openshift-controller-manager, packageserver, machine-approver, cluster-storage-operator, cluster-version-operator, cluster-policy-controller, openshift-route-controller-manager, community-operators-catalog, cluster-image-registry-operator, dns-operator, konnectivity-agent, olm-operator, azure-cloud-controller-manager, csi-snapshot-controller-operator, hosted-cluster-config-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
            correlation ID: a21d9aa7-da4e-4f79-9da1-353faa5bb8d0
            client request ID: 457b6d19-40ff-4036-a44f-729e35535125
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4jxfrx" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3b2f64940d00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4jxfrx: failed waiting for cluster="cp-ystream-upgrade-4-21-4jxfrx" in resourcegroup="rg-cp-ystream-upgrade-4-21-dxmr9r2kr8sj" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "name": "e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:38.502730447Z",
      "endTime": "2026-10-19T11:00:44.46660972Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 23m51s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
            correlation ID: 72780af9-37e8-42d0-b405-3ed5aea87756
            client request ID: 04371127-7fbb-474b-a1aa-c21583ae4e63
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4jxfrx" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3b2f64940d00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4jxfrx: failed waiting for cluster="cp-ystream-upgrade-4-21-4jxfrx" in resourcegroup="rg-cp-ystream-upgrade-4-21-dxmr9r2kr8sj" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "name": "e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:38.502730447Z",
      "endTime": "2026-10-19T11:00:44.46660972Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 23m51s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
            correlation ID: 72780af9-37e8-42d0-b405-3ed5aea87756
            client request ID: 04371127-7fbb-474b-a1aa-c21583ae4e63
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-ld4f2qxz9tlh/cluster-candidate-4-23-prg9qx should provision
Unexpected error:
    <*fmt.wrapError | 0x103e49a6a6a0>: 
    failed to create HCP cluster cluster-candidate-4-23-prg9qx: failed waiting for cluster="cluster-candidate-4-23-prg9qx" in resourcegroup="rg-candidate-4-23-ld4f2qxz9tlh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900",
      "name": "87105fe8-f597-4050-9513-e5fec5cf8900",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.32598821Z",
      "endTime": "2026-10-19T11:00:34.343098485Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, cluster-autoscaler, cluster-image-registry-operator, redhat-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, catalog-operator, kube-storage-version-migrator, hosted-cluster-config-operator, olm-collect-profiles, konnectivity-agent, kube-scheduler, machine-approver, cluster-storage-operator, certified-operators-catalog, cluster-node-tuning-operator, community-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, kube-controller-manager, olm-operator, control-plane-pki-operator, ingress-operator, cluster-version-operator, openshift-apiserver, cluster-policy-controller, dns-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
            correlation ID: fc208c85-0c22-4fd2-b0d8-868115a21156
            client request ID: 2c46da4c-f387-4bd8-8b83-56d49c4e1b76
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-ld4f2qxz9tlh/cluster-candidate-4-23-prg9qx should provision
Unexpected error:
    <*fmt.wrapError | 0x103e49a6a6a0>: 
    failed to create HCP cluster cluster-candidate-4-23-prg9qx: failed waiting for cluster="cluster-candidate-4-23-prg9qx" in resourcegroup="rg-candidate-4-23-ld4f2qxz9tlh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900",
      "name": "87105fe8-f597-4050-9513-e5fec5cf8900",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.32598821Z",
      "endTime": "2026-10-19T11:00:34.343098485Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, cluster-autoscaler, cluster-image-registry-operator, redhat-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, catalog-operator, kube-storage-version-migrator, hosted-cluster-config-operator, olm-collect-profiles, konnectivity-agent, kube-scheduler, machine-approver, cluster-storage-operator, certified-operators-catalog, cluster-node-tuning-operator, community-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, kube-controller-manager, olm-operator, control-plane-pki-operator, ingress-operator, cluster-version-operator, openshift-apiserver, cluster-policy-controller, dns-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
            correlation ID: fc208c85-0c22-4fd2-b0d8-868115a21156
            client request ID: 2c46da4c-f387-4bd8-8b83-56d49c4e1b76
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x1eb3701d2840>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ljv8hq9fnscx" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324376be-383b-4d90-8413-08396178b495
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324376be-383b-4d90-8413-08396178b495",
      "name": "324376be-383b-4d90-8413-08396178b495",
      "status": "Failed",
      "startTime": "2026-10-19T10:33:06.977502829Z",
      "endTime": "2026-10-19T11:02:14.171233214Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324376be-383b-4d90-8413-08396178b495
            correlation ID: f6af9da5-5eba-49f3-9f86-e6e1124bd7f3
            client request ID: 30b2ba1c-045a-4d02-9c28-804fd3a69b7f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x1eb3701d2840>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ljv8hq9fnscx" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324376be-383b-4d90-8413-08396178b495
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324376be-383b-4d90-8413-08396178b495",
      "name": "324376be-383b-4d90-8413-08396178b495",
      "status": "Failed",
      "startTime": "2026-10-19T10:33:06.977502829Z",
      "endTime": "2026-10-19T11:02:14.171233214Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324376be-383b-4d90-8413-08396178b495
            correlation ID: f6af9da5-5eba-49f3-9f86-e6e1124bd7f3
            client request ID: 30b2ba1c-045a-4d02-9c28-804fd3a69b7f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-bkqnpw2l8zf5/cluster-candidate-5-0-ccpvpc should provision
Unexpected error:
    <*fmt.wrapError | 0x3d23d4310dc0>: 
    failed to create HCP cluster cluster-candidate-5-0-ccpvpc: failed waiting for cluster="cluster-candidate-5-0-ccpvpc" in resourcegroup="rg-candidate-5-0-bkqnpw2l8zf5" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c",
      "name": "44c0dc0d-efb0-45b9-885c-ff622849913c",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:41.722981857Z",
      "endTime": "2026-10-19T11:00:44.465792588Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
            correlation ID: 77467a09-84cb-420c-a3bc-027f97f67194
            client request ID: 2b188110-77bd-4bc4-9d94-27a812b7e09a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-bkqnpw2l8zf5/cluster-candidate-5-0-ccpvpc should provision
Unexpected error:
    <*fmt.wrapError | 0x3d23d4310dc0>: 
    failed to create HCP cluster cluster-candidate-5-0-ccpvpc: failed waiting for cluster="cluster-candidate-5-0-ccpvpc" in resourcegroup="rg-candidate-5-0-bkqnpw2l8zf5" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c",
      "name": "44c0dc0d-efb0-45b9-885c-ff622849913c",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:41.722981857Z",
      "endTime": "2026-10-19T11:00:44.465792588Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
            correlation ID: 77467a09-84cb-420c-a3bc-027f97f67194
            client request ID: 2b188110-77bd-4bc4-9d94-27a812b7e09a
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xbe67dc5ca00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-gs2cw8rqvtsp" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "name": "9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "status": "Failed",
      "startTime": "2026-10-19T10:33:35.601658411Z",
      "endTime": "2026-10-19T11:02:44.344881038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, csi-snapshot-controller-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, ingress-operator, olm-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, kube-controller-manager, openshift-controller-manager, cluster-network-operator, redhat-marketplace-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-policy-controller, azure-cloud-controller-manager, packageserver, cluster-version-operator, openshift-apiserver, cluster-autoscaler, machine-approver, capi-provider, catalog-operator, community-operators-catalog, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 2 unavailable replicas, kube-controller-manager deployment has 2 unavailable replicas, kube-scheduler deployment has 2 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
            correlation ID: ebcf2a7d-e9a4-4dbb-b982-a31f966e3404
            client request ID: d11643b0-b580-4f4f-a16e-c60b01f474c0
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xbe67dc5ca00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-gs2cw8rqvtsp" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "name": "9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "status": "Failed",
      "startTime": "2026-10-19T10:33:35.601658411Z",
      "endTime": "2026-10-19T11:02:44.344881038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, csi-snapshot-controller-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, ingress-operator, olm-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, kube-controller-manager, openshift-controller-manager, cluster-network-operator, redhat-marketplace-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-policy-controller, azure-cloud-controller-manager, packageserver, cluster-version-operator, openshift-apiserver, cluster-autoscaler, machine-approver, capi-provider, catalog-operator, community-operators-catalog, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 2 unavailable replicas, kube-controller-manager deployment has 2 unavailable replicas, kube-scheduler deployment has 2 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
            correlation ID: ebcf2a7d-e9a4-4dbb-b982-a31f966e3404
            client request ID: d11643b0-b580-4f4f-a16e-c60b01f474c0
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x22a4ab41e440>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-tkvqmscnx8hd" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25",
      "name": "7966e350-c6be-4e91-9c2e-1655783cec25",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:11.987681126Z",
      "endTime": "2026-10-19T11:01:14.305309512Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
            correlation ID: 8f412970-2932-49aa-85c1-e501fdb88128
            client request ID: 037bff49-0293-4f3d-ab7f-9ff8b71e3d4a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x22a4ab41e440>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-tkvqmscnx8hd" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25",
      "name": "7966e350-c6be-4e91-9c2e-1655783cec25",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:11.987681126Z",
      "endTime": "2026-10-19T11:01:14.305309512Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
            correlation ID: 8f412970-2932-49aa-85c1-e501fdb88128
            client request ID: 037bff49-0293-4f3d-ab7f-9ff8b71e3d4a
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x22128cbeb10>: 
    GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "name": "26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:37.259328289Z",
      "endTime": "2026-10-19T11:01:44.306693815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-node-tuning-operator, certified-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, dns-operator, ingress-operator, redhat-operators-catalog, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
            correlation ID: 87b43167-8612-41cf-b2af-01e82520620b
            client request ID: 826cfb23-3364-490c-b9f1-549ca623afb4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x22128cbeb10>: 
    GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "name": "26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:37.259328289Z",
      "endTime": "2026-10-19T11:01:44.306693815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-node-tuning-operator, certified-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, dns-operator, ingress-operator, redhat-operators-catalog, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
            correlation ID: 87b43167-8612-41cf-b2af-01e82520620b
            client request ID: 826cfb23-3364-490c-b9f1-549ca623afb4
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:136]: failed to verify HCP cluster "cluster-candidate-4-21-d6zdrd" is viable
Unexpected error:
    <*errors.joinError | 0xde74c1ac0a8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xde74c2c7be0>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xde74c2c7bc0>{
                    msg: "failed to list services: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xde74be74ff0>{
                        Op: "Get",
                        URL: "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xde74bbe7a90>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xde74b874e10>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                                Name: "api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xde74b4feb80>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xde74b4feb40>{
                    msg: "failed to list all APIServices: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xde74be1b410>{
                        Op: "Get",
                        URL: "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xde74b874fa0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xde74b874e10>0xde74b874e10 (cyclic reference),
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:136]: failed to verify HCP cluster "cluster-candidate-4-21-d6zdrd" is viable
Unexpected error:
    <*errors.joinError | 0xde74c1ac0a8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xde74c2c7be0>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xde74c2c7bc0>{
                    msg: "failed to list services: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xde74be74ff0>{
                        Op: "Get",
                        URL: "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xde74bbe7a90>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xde74b874e10>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                                Name: "api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xde74b4feb80>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xde74b4feb40>{
                    msg: "failed to list all APIServices: Get \"https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.96.92:35750->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xde74be1b410>{
                        Op: "Get",
                        URL: "https://api.t5f2q1v9y1k8g4h.3m43.j7965568.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xde74b874fa0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xde74b874e10>0xde74b874e10 (cyclic reference),
                        },
                    },
                },
            },
        ],
    }
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T11:18:35Z
  Ended:       2026-10-19T11:31:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/capi-provider is not progressing for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T11:18:35Z
  Ended:       2026-10-19T11:31:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/capi-provider is not progressing for longer than 30 minutes.

alert fired 26 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:08:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:08:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="set-kubelet-parameters-for-scale", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/set-kubelet-parameters-for-scale has not finished or progressed for at least 15 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:08:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="set-kubelet-parameters-for-scale", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/set-kubelet-parameters-for-scale has not finished or progressed for at least 15 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:08:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-19T10:56:20Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-19T10:56:21Z
  Ended:       2026-10-19T11:08:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-19T11:01:19Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-19T11:01:19Z
  Ended:       2026-10-19T11:07:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T10:56:32Z
  Ended:       2026-10-19T11:24:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1bb593ff-9c5b-45f3-a6e4-58b2cc1e759f"
  Description: Pod ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T10:56:32Z
  Ended:       2026-10-19T11:24:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1bb593ff-9c5b-45f3-a6e4-58b2cc1e759f"
  Description: Pod ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T10:57:11Z
  Ended:       2026-10-19T11:07:12Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="prometheus", node="aks-userswft1-30082608-vmss000000", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-30082608-vmss000000 is unreachable and some workloads may be rescheduled.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T10:57:11Z
  Ended:       2026-10-19T11:07:12Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="prometheus", node="aks-userswft1-30082608-vmss000000", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-30082608-vmss000000 is unreachable and some workloads may be rescheduled.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T10:56:23Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T10:56:23Z
  Ended:       2026-10-19T11:07:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/services-j7965568", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.

alert fired 66 time(s) (42 unknown, 24 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T10:44:35Z
  Ended:       2026-10-19T10:51:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7oomta392aplhs9dcvdp28vtt5he-cilium-cluster", pod="etcd-1", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7oomta392aplhs9dcvdp28vtt5he-cilium-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T10:44:35Z
  Ended:       2026-10-19T10:51:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas", pod="etcd-1", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T10:44:35Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq800b8r02hfj17953dpa5qhvqgvb7-agg-cluster", pod="etcd-1", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq800b8r02hfj17953dpa5qhvqgvb7-agg-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="cluster-policy-controller-8889f97bf-twmrh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/cluster-policy-controller-8889f97bf-twmrh has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7404tc5dseflpaqmlgdjmpplinrv-s8a1x8h4i2l2k9y", pod="kube-apiserver-85c8fd75b9-pj5z4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7404tc5dseflpaqmlgdjmpplinrv-s8a1x8h4i2l2k9y/kube-apiserver-85c8fd75b9-pj5z4 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="openshift-route-controller-manager-6f94c89644-rpn2h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/openshift-route-controller-manager-6f94c89644-rpn2h has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="csi-snapshot-controller-operator-565fb6b897-52hk6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/csi-snapshot-controller-operator-565fb6b897-52hk6 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="cluster-network-operator-6c88f69f48-9nb2d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/cluster-network-operator-6c88f69f48-9nb2d has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="cluster-storage-operator-7775d5d868-bp7g4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/cluster-storage-operator-7775d5d868-bp7g4 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster", pod="kube-scheduler-86dcc8bf4f-qpbrh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster/kube-scheduler-86dcc8bf4f-qpbrh has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u", pod="kube-apiserver-7f8bf74cb4-tvz2n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u/kube-apiserver-7f8bf74cb4-tvz2n has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="control-plane-pki-operator-7ff86fdd89-d56mn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/control-plane-pki-operator-7ff86fdd89-d56mn has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="cluster-image-registry-operator-6646df896c-cj5wk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/cluster-image-registry-operator-6646df896c-cj5wk has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="ingress-operator-65df885c85-5plmd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/ingress-operator-65df885c85-5plmd has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="openshift-controller-manager-5b799fbc74-wklbs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/openshift-controller-manager-5b799fbc74-wklbs has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="cluster-version-operator-665989cf9-tgztw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/cluster-version-operator-665989cf9-tgztw has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="dns-operator-778b6fd76c-ljzf6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/dns-operator-778b6fd76c-ljzf6 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="cluster-node-tuning-operator-55b75b94dc-6tcrk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/cluster-node-tuning-operator-55b75b94dc-6tcrk has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="cluster-network-operator-564c8cd855-j6ww2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/cluster-network-operator-564c8cd855-j6ww2 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="cluster-policy-controller-5c86cfdd85-nn7lg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/cluster-policy-controller-5c86cfdd85-nn7lg has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="cluster-storage-operator-56885bd74c-xwqvp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/cluster-storage-operator-56885bd74c-xwqvp has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster", pod="openshift-apiserver-658c9854f7-tszzr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster/openshift-apiserver-658c9854f7-tszzr has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="kube-controller-manager-7667c986c-hdvfl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/kube-controller-manager-7667c986c-hdvfl has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-19T10:45:34Z
  Ended:       2026-10-19T10:51:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="cluster-version-operator-6569cb855c-kzf2p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/cluster-version-operator-6569cb855c-kzf2p has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:23:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas", pod="router-c5bd85547-nwn6l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas/router-c5bd85547-nwn6l has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="openshift-apiserver-86cb9c89c-jhkcd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/openshift-apiserver-86cb9c89c-jhkcd has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:08:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u", pod="kube-apiserver-7f8bf74cb4-fccnx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u/kube-apiserver-7f8bf74cb4-fccnx has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq800b8r02hfj17953dpa5qhvqgvb7-agg-cluster", pod="router-5f55f7b569-tslff", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq800b8r02hfj17953dpa5qhvqgvb7-agg-cluster/router-5f55f7b569-tslff has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7oomta392aplhs9dcvdp28vtt5he-cilium-cluster", pod="router-6959d5b854-9hrjr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7oomta392aplhs9dcvdp28vtt5he-cilium-cluster/router-6959d5b854-9hrjr has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="openshift-apiserver-7ffff95896-fz5vn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/openshift-apiserver-7ffff95896-fz5vn has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-19T10:51:32Z
  Ended:       2026-10-19T11:19:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="router-6c69dcc6c6-wth6h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/router-6c69dcc6c6-wth6h has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p", pod="kube-apiserver-7689969d44-txkvr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7h85q1ncqh2sbc0rd81ub3s5shc9-g0j4a0v8o8g8t8p/kube-apiserver-7689969d44-txkvr has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq72ld0ojj1mp3uk1lgmrffmlrl99b-l3v8s2l3c9m7o0r", pod="network-node-identity-6586679c4c-pcpd9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq72ld0ojj1mp3uk1lgmrffmlrl99b-l3v8s2l3c9m7o0r/network-node-identity-6586679c4c-pcpd9 has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster", pod="openshift-apiserver-658c9854f7-vgvdp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster/openshift-apiserver-658c9854f7-vgvdp has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq72ld0ojj1mp3uk1lgmrffmlrl99b-l3v8s2l3c9m7o0r", pod="openshift-apiserver-858b75549d-2j2rr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq72ld0ojj1mp3uk1lgmrffmlrl99b-l3v8s2l3c9m7o0r/openshift-apiserver-858b75549d-2j2rr has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics", pod="kube-apiserver-56967ffcb8-v4xbp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7b23klftd67ff71s751eboi9kk1p-ksm-hcp-metrics/kube-apiserver-56967ffcb8-v4xbp has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:08:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq72tljm8flvc8illdor05q2d3jsvu-q1z3g1k8p2m8q3s", pod="kube-apiserver-5fbf5fd755-w7kwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq72tljm8flvc8illdor05q2d3jsvu-q1z3g1k8p2m8q3s/kube-apiserver-5fbf5fd755-w7kwc has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:18:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u", pod="router-59cf5997f6-nzxvq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u/router-59cf5997f6-nzxvq has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq72tljm8flvc8illdor05q2d3jsvu-q1z3g1k8p2m8q3s", pod="openshift-apiserver-7569f5c7df-dzhsx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq72tljm8flvc8illdor05q2d3jsvu-q1z3g1k8p2m8q3s/openshift-apiserver-7569f5c7df-dzhsx has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:08:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq7404tc5dseflpaqmlgdjmpplinrv-s8a1x8h4i2l2k9y", pod="kube-apiserver-85c8fd75b9-x8l99", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq7404tc5dseflpaqmlgdjmpplinrv-s8a1x8h4i2l2k9y/kube-apiserver-85c8fd75b9-x8l99 has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster", pod="kube-apiserver-985c6d9f9-cs7rf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq738p1bcdqbp8b2i3gbn005aaui84-basic-cluster/kube-apiserver-985c6d9f9-cs7rf has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-19T10:51:33Z
  Ended:       2026-10-19T11:08:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thq72ld0ojj1mp3uk1lgmrffmlrl99b-l3v8s2l3c9m7o0r", pod="kube-apiserver-6ddbf8f745-8tgg9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2thq72ld0ojj1mp3uk1lgmrffmlrl99b-l3v8s2l3c9m7o0r/kube-apiserver-6ddbf8f745-8tgg9 has been in a non-ready state for longer than 5 minutes.

alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T11:08:39Z
  Ended:       2026-10-19T11:31:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T11:08:39Z
  Ended:       2026-10-19T11:31:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T11:11:35Z
  Ended:       2026-10-19T11:19:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T11:11:35Z
  Ended:       2026-10-19T11:18:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T11:11:35Z
  Ended:       2026-10-19T11:19:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T11:11:35Z
  Ended:       2026-10-19T11:23:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq717si3aet42im2u3mgm1p94lgs83-private-kas/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T11:11:35Z
  Ended:       2026-10-19T11:18:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq70r9o3n24hq38t4ueapic16a3d74-a3v0z7g8c2d5e0u/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T11:11:35Z
  Ended:       2026-10-19T11:19:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7965568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7965568/providers/microsoft.monitor/accounts/hcps-j7965568", namespace="ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mmhvv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thq73tilorccuv8p9876u7h0dp8pm57-j9m4x7q3m8f8l0e/router has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T09:12:06Z
  Ended:       2026-10-19T09:30:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1399168-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x59ce69d41c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x59ce69eea20>{
            s: "dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x59ce69d41c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x59ce69eea20>{
            s: "dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:373]: expected 2 ready and schedulable nodes after the cordon/drain/uncordon cycle
Expected success, but got an error:
    <*errors.errorString | 0x3f000f2687f0>: 
    expected 2 ready (and schedulable) nodes in node pool "cordon-np", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]
    {
        s: "expected 2 ready (and schedulable) nodes in node pool \"cordon-np\", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:373]: expected 2 ready and schedulable nodes after the cordon/drain/uncordon cycle
Expected success, but got an error:
    <*errors.errorString | 0x3f000f2687f0>: 
    expected 2 ready (and schedulable) nodes in node pool "cordon-np", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]
    {
        s: "expected 2 ready (and schedulable) nodes in node pool \"cordon-np\", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:184]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0x1e0c6d55a40>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1e0c6d55a20>{
            msg: "failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1e0c6d55a00>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:184]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0x1e0c6d55a40>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1e0c6d55a20>{
            msg: "failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1e0c6d55a00>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:519]: Timed out after 900.005s.
node labels/taints should appear on the intended node pools of cluster "agg-cluster"
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:506 with:
node pool "agg-np-a" should have at least 2 nodes (autoscaling min), found 0
Expected
    <int>: 0
to be >=
    <int>: 2
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:519]: Timed out after 900.005s.
node labels/taints should appear on the intended node pools of cluster "agg-cluster"
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:506 with:
node pool "agg-np-a" should have at least 2 nodes (autoscaling min), found 0
Expected
    <int>: 0
to be >=
    <int>: 2

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T08:12:56Z
  Ended:       2026-10-19T08:32:56Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1968640-svc", component="backend", name="csstatedump", region="centralus", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T08:12:56Z
  Ended:       2026-10-19T08:31:55Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1968640-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-10-19T07:33:05.070Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T07:33:05.599Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T07:33:05.599Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T07:33:05.710Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-19T07:33:05.849Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-19T07:33:05.849Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T07:33:05.925Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T07:33:56.641Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T07:33:56.642Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x54e9aa24000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-4wl22sb8l6px" to finish creating: GET https://rp.j6052224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47d30f4f-4041-41a8-ac8b-37f3547ef206
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47d30f4f-4041-41a8-ac8b-37f3547ef206",
      "name": "47d30f4f-4041-41a8-ac8b-37f3547ef206",
      "status": "Failed",
      "startTime": "2026-10-19T04:11:42.972869191Z",
      "endTime": "2026-10-19T04:30:45.485858922Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-d4j2g-r4b85: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-4wl22sb8l6px--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-d4j2g-r4b85 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-d4j2g-r4b85: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-4wl22sb8l6px--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-d4j2g-r4b85 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6052224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47d30f4f-4041-41a8-ac8b-37f3547ef206
            correlation ID: 2b14871c-7eb2-49b8-a678-9dcdb4e96cad
            client request ID: aa1ac31a-55fd-4750-9293-aaaca548fae3
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x54e9aa24000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-4wl22sb8l6px" to finish creating: GET https://rp.j6052224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47d30f4f-4041-41a8-ac8b-37f3547ef206
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47d30f4f-4041-41a8-ac8b-37f3547ef206",
      "name": "47d30f4f-4041-41a8-ac8b-37f3547ef206",
      "status": "Failed",
      "startTime": "2026-10-19T04:11:42.972869191Z",
      "endTime": "2026-10-19T04:30:45.485858922Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-d4j2g-r4b85: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-4wl22sb8l6px--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-d4j2g-r4b85 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-d4j2g-r4b85: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-4wl22sb8l6px--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-d4j2g-r4b85 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6052224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47d30f4f-4041-41a8-ac8b-37f3547ef206
            correlation ID: 2b14871c-7eb2-49b8-a678-9dcdb4e96cad
            client request ID: aa1ac31a-55fd-4750-9293-aaaca548fae3