CIHealth
<
Single Day: Oct 7▾
Last 7 Days Weekly: Oct 5 - Oct 11 Sprint: Sep 28 - Oct 11 Single Day: Oct 7
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 51 · Failures matched: 425 · Total runs: 71 · Runs affected: 55

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e22.82%2 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x295f3df44be0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-f56729plnfx5": PUT https://rp.j6769152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-f56729plnfx5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x295f3df44be0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-f56729plnfx5": PUT https://rp.j6769152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-f56729plnfx5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x156cf26d7900>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-j4cvjbg6jcxh": PUT https://rp.j0287232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j4cvjbg6jcxh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x156cf26d7900>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-j4cvjbg6jcxh": PUT https://rp.j0287232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j4cvjbg6jcxh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73972107751278686769152
PR #73972107681070840287232
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to create nodepool <nodepool>: failed starting nodepool creation "np-1" for cluster "<cluster>" in resourcegroup="<resource-group>": PUT <url>
full failure pattern:
failed to create nodepool <nodepool>: failed starting nodepool creation "np-1" for cluster "<cluster>" in resourcegroup="<resource-group>": PUT <url>
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e22.82%2 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e182923c240>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ls445v45mtp4": PUT https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-ls445v45mtp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e182923c240>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ls445v45mtp4": PUT https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-ls445v45mtp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:162]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e9eea24a6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fh65w65xjrlv": PUT https://rp.j8091008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fh65w65xjrlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:162]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e9eea24a6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fh65w65xjrlv": PUT https://rp.j8091008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fh65w65xjrlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.12
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73252107916955439599616
PR #73252107817608098091008
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
full failure pattern:
ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:142]: Timed out after 1200.007s.
Insights Operator did not reach Disabled/NoToken state, the NoToken-to-healthy transition cannot be tested
Expected success, but got an error:
    <*errors.errorString | 0x27811265a7d0>: 
    ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
    {
        s: "ClusterOperator \"insights\" not yet in Disabled/NoToken state (conditions: [])",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:142]: Timed out after 1200.007s.
Insights Operator did not reach Disabled/NoToken state, the NoToken-to-healthy transition cannot be tested
Expected success, but got an error:
    <*errors.errorString | 0x27811265a7d0>: 
    ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
    {
        s: "ClusterOperator \"insights\" not yet in Disabled/NoToken state (conditions: [])",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to see an HCP cluster registered with console.redhat.com via the Insights Operator1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69542107939651057618944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3974085580d8>: 
    resource group "external-auth-cluster-jkpp8mdvzxsh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-jkpp8mdvzxsh" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac",
      "name": "7ef42875-c74c-425c-8fe8-0909778dceac",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:50.350274496Z",
      "endTime": "2026-10-07T18:24:57.82988818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3adt66dtiffarpvt6smilc2f19037 still exists (deletion dispatched at 2026-10-07T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3974085580d8>: 
    resource group "external-auth-cluster-jkpp8mdvzxsh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-jkpp8mdvzxsh" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac",
      "name": "7ef42875-c74c-425c-8fe8-0909778dceac",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:50.350274496Z",
      "endTime": "2026-10-07T18:24:57.82988818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3adt66dtiffarpvt6smilc2f19037 still exists (deletion dispatched at 2026-10-07T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71432107875990943305728
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2183e1e34210>: 
    resource group "private-ingress-76lkp46mpc8d": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-76lkp46mpc8d" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "name": "1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "status": "Failed",
      "startTime": "2026-10-07T18:03:40.469898232Z",
      "endTime": "2026-10-07T18:27:41.300163086Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cd0prbd7pkt4n50qh2rsnhp0bjo4 still exists (deletion dispatched at 2026-10-07T18:03:41Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2183e1e34210>: 
    resource group "private-ingress-76lkp46mpc8d": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-76lkp46mpc8d" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "name": "1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "status": "Failed",
      "startTime": "2026-10-07T18:03:40.469898232Z",
      "endTime": "2026-10-07T18:27:41.300163086Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cd0prbd7pkt4n50qh2rsnhp0bjo4 still exists (deletion dispatched at 2026-10-07T18:03:41Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74002107875420446658560
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f397df201c8>: 
    resource group "clusternp128-89h59bcbsmnq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-89h59bcbsmnq" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "name": "84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:09.863415259Z",
      "endTime": "2026-10-07T18:24:11.307794702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3chch61ipf94injnrljj1hk0caf92 still exists (deletion dispatched at 2026-10-07T18:00:10Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f397df201c8>: 
    resource group "clusternp128-89h59bcbsmnq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-89h59bcbsmnq" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "name": "84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:09.863415259Z",
      "endTime": "2026-10-07T18:24:11.307794702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3chch61ipf94injnrljj1hk0caf92 still exists (deletion dispatched at 2026-10-07T18:00:10Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74002107875420446658560
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderCluster...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xeda890f47f8>: 
    resource group "oidc-wi-h74rjsb4b6b4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-h74rjsb4b6b4" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98",
      "name": "347d2fde-a5b9-4245-8439-df8aaab88f98",
      "status": "Failed",
      "startTime": "2026-10-07T18:11:44.745635305Z",
      "endTime": "2026-10-07T18:35:51.763179185Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xeda890f47f8>: 
    resource group "oidc-wi-h74rjsb4b6b4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-h74rjsb4b6b4" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98",
      "name": "347d2fde-a5b9-4245-8439-df8aaab88f98",
      "status": "Failed",
      "startTime": "2026-10-07T18:11:44.745635305Z",
      "endTime": "2026-10-07T18:35:51.763179185Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71432107875990943305728
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRInternal error occurred: failed calling webhook "webhook
full failure pattern:
Internal error occurred: failed calling webhook "webhook
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:337]: Timed out after 120.000s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
    <*errors.StatusError | 0x3dfc585d0640>: 
    Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:337]: Timed out after 120.000s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
    <*errors.StatusError | 0x3dfc585d0640>: 
    Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73702107750168211230720
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-<id> on node <node> after retries: a...
full failure pattern:
VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-<id> on node <node> after retries: an error on the server ("unknown") has prevented the request from succeeding
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xf6f46a980d8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-tkqcj)
    {
        errs: [
            <*fmt.wrapError | 0xf6f45174120>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                err: <*fmt.wrapError | 0xf6f45174100>{
                    msg: "failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                    err: <*fmt.wrapError | 0xf6f451742a0>{
                        msg: "failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                        err: <*errors.StatusError | 0xf6f4458f7c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-tkqcj",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xf6f46a980d8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-tkqcj)
    {
        errs: [
            <*fmt.wrapError | 0xf6f45174120>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                err: <*fmt.wrapError | 0xf6f45174100>{
                    msg: "failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                    err: <*fmt.wrapError | 0xf6f451742a0>{
                        msg: "failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                        err: <*errors.StatusError | 0xf6f4458f7c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-tkqcj",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73252107916955439599616
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyHyperShiftIngressCertificate timed out after 10m0
full failure pattern:
VerifyHyperShiftIngressCertificate timed out after 10m0
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73252107970662109810688
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...
full failure pattern:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0x339d2c183220>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);
    let _endTime = datetime(2026-10-07T22:32:32.2954057Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-n8d6fgpt4qrq'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T22:32:43.6043688Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd", ActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", SubActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityType:"GW.Http.CallContext", ParentActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityStack:"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x339d2b3c97a0>{
            errs: [
                <*fmt.wrapError | 0x339d2c183200>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x339d2c1831e0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x339d2c1830c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x339d2c183080>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x339d2ae3d200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0x339d2c183220>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);
    let _endTime = datetime(2026-10-07T22:32:32.2954057Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-n8d6fgpt4qrq'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T22:32:43.6043688Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd", ActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", SubActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityType:"GW.Http.CallContext", ParentActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityStack:"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x339d2b3c97a0>{
            errs: [
                <*fmt.wrapError | 0x339d2c183200>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x339d2c1831e0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x339d2c1830c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x339d2c183080>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x339d2ae3d200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69542107939651057618944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...
full failure pattern:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.006s.
Expected success, but got an error:
    <*fmt.wrapError | 0x11eea31d1d40>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);
    let _endTime = datetime(2026-10-08T00:23:30.3337686Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7nt846zks6zp'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T00:23:37.3079994Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d", ActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", SubActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityType:"GW.Http.CallContext", ParentActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityStack:"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x11eea2518d38>{
            errs: [
                <*fmt.wrapError | 0x11eea31d1d20>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x11eea31d1d00>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x11eea2cd05c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x11eea2cd0560>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x11eea1622000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.006s.
Expected success, but got an error:
    <*fmt.wrapError | 0x11eea31d1d40>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);
    let _endTime = datetime(2026-10-08T00:23:30.3337686Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7nt846zks6zp'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T00:23:37.3079994Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d", ActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", SubActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityType:"GW.Http.CallContext", ParentActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityStack:"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x11eea2518d38>{
            errs: [
                <*fmt.wrapError | 0x11eea31d1d20>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x11eea31d1d00>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x11eea2cd05c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x11eea2cd0560>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x11eea1622000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69282107969278706716672
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline...
full failure pattern:
failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision11.41%1 of 71 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
time=2026-10-07T14:40:32.913Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-10-07T14:40:33.412Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157 global:map[imageOverrides:map[governance_policy_propagator:governance-policy-propagator-acm-216@sha256:deb737b19932d520553c902ee5e0738ebeaad4592e8aa9649c41fec6fc00b62e] registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-10-07T14:40:33.412Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:40:33.440Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 namespace=multicluster-engine
time=2026-10-07T14:40:33.491Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 dryRun=false
time=2026-10-07T14:40:33.491Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:40:33.534Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:51:01.640Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 err="failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-10-07T14:51:01.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config (stamp=1)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972107832305794945024
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
alert2940.85%29 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsOct 1: 5 · Oct 2: 19 · Oct 3: 15 · Oct 4: 3 · Oct 5: 26 · Oct 6: 15 · Oct 7: 29none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T00:05:01Z
  Ended:       2026-10-08T00:11:00Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6670976-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T23:44:13Z
  Ended:       2026-10-07T23:58:13Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6213760-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T21:16:57Z
  Ended:       2026-10-07T21:22:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3227776-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire29
Affected runs (29)
Date (UTC)Associated PRProw job
PR #74172107967978476670976
PR #74142107964840596213760
PR #63642107926490833227776
PR #73052107926353130033152
PR #74142107916465603612672
PR #74122107914072107257856
PR #72882107904235558408192
PR #72482107883389313880064
PR #68072107880778506440704
PR #71432107875990943305728
PR #73222107875761300967424
PR #74002107875420446658560
PR #73102107872947002675200
PR #74102107844183950626816
PR #74042107834229550223360
PR #63642107828301232869376
PR #74052107802095104036864
PR #66882107778619966230528
PR #71332107752436901875712
PR #74032107751899905134592
PR #73972107751278686769152
PR #71432107750854802018304
PR #73992107750792218808320
PR #73712107750686056779776
PR #73702107750168211230720
PR #69682107750153682161664
PR #73102107739528360890368
PR #73972107681070840287232
PR #70972107680144293367808
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e1723.94%17 of 71 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsOct 1: 21 · Oct 2: 21 · Oct 3: 10 · Oct 4: 61 · Oct 5: 276 · Oct 6: 236 · Oct 7: 126PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x2884a5492ee0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dk2zvdznnrzn" to finish creating: GET https://rp.j7618944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "name": "9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "status": "Failed",
      "startTime": "2026-10-07T22:28:55.765188822Z",
      "endTime": "2026-10-07T22:57:56.06793791Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x2884a5492ee0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dk2zvdznnrzn" to finish creating: GET https://rp.j7618944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "name": "9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "status": "Failed",
      "startTime": "2026-10-07T22:28:55.765188822Z",
      "endTime": "2026-10-07T22:57:56.06793791Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm"
Unexpected error:
    <*fmt.wrapError | 0x2aa4ed1a5b40>: 
    failed to create NodePool np-ckh4xm: failed waiting for nodepool="np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm" in resourcegroup="rg-candidate-4-23-g2p8crd8t6kf" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7",
      "name": "7981efbe-9393-4366-bd69-e17288ac49a7",
      "status": "Failed",
      "startTime": "2026-10-07T20:35:08.39379338Z",
      "endTime": "2026-10-07T21:04:16.844016594Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm"
Unexpected error:
    <*fmt.wrapError | 0x2aa4ed1a5b40>: 
    failed to create NodePool np-ckh4xm: failed waiting for nodepool="np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm" in resourcegroup="rg-candidate-4-23-g2p8crd8t6kf" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7",
      "name": "7981efbe-9393-4366-bd69-e17288ac49a7",
      "status": "Failed",
      "startTime": "2026-10-07T20:35:08.39379338Z",
      "endTime": "2026-10-07T21:04:16.844016594Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq"
Unexpected error:
    <*fmt.wrapError | 0x99848406080>: 
    failed to create NodePool np-mcx5bq: failed waiting for nodepool="np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq" in resourcegroup="rg-candidate-5-1-tvjcv9r9dthd" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d",
      "name": "fe8bd558-9f50-442b-bfc7-12311927703d",
      "status": "Failed",
      "startTime": "2026-10-07T18:23:04.012039216Z",
      "endTime": "2026-10-07T18:52:11.882573758Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq"
Unexpected error:
    <*fmt.wrapError | 0x99848406080>: 
    failed to create NodePool np-mcx5bq: failed waiting for nodepool="np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq" in resourcegroup="rg-candidate-5-1-tvjcv9r9dthd" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d",
      "name": "fe8bd558-9f50-442b-bfc7-12311927703d",
      "status": "Failed",
      "startTime": "2026-10-07T18:23:04.012039216Z",
      "endTime": "2026-10-07T18:52:11.882573758Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (20)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion13
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.2213
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.012
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.2111
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.111
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.2010
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public10
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.239
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with a private key vault using an available API version1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a fully private cluster with private KAS and private ingress and verify both are only reachable from VNet1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a fully private cluster with private KAS, private ingress, and private KeyVault and verify all are operational1
Affected runs (17)
Date (UTC)Associated PRProw job
PR #69542107939651057618944
PR #73252107916955439599616
PR #72482107883389313880064
PR #68072107880778506440704
PR #74142107879778638893056
PR #71432107875990943305728
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #66882107778619966230528
PR #74052107765914286952448
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
PR #73102107739528360890368
PR #73972107681070840287232
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
e2e1216.90%12 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 1: 0 · Oct 2: 4 · Oct 3: 0 · Oct 4: 5 · Oct 5: 36 · Oct 6: 39 · Oct 7: 16PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x259bbf69a258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6m6qldl4t7ln" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "name": "74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "status": "Failed",
      "startTime": "2026-10-07T18:16:07.02818237Z",
      "endTime": "2026-10-07T18:45:11.860346136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x259bbf69a258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6m6qldl4t7ln" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "name": "74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "status": "Failed",
      "startTime": "2026-10-07T18:16:07.02818237Z",
      "endTime": "2026-10-07T18:45:11.860346136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x14bbdf4ea0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-db8t5zc8d27b" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b",
      "name": "28c778fc-2439-43ac-8817-96efd7f98d9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:23.837365353Z",
      "endTime": "2026-10-07T18:49:29.442401902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x14bbdf4ea0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-db8t5zc8d27b" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b",
      "name": "28c778fc-2439-43ac-8817-96efd7f98d9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:23.837365353Z",
      "endTime": "2026-10-07T18:49:29.442401902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features9
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully7
Affected runs (12)
Date (UTC)Associated PRProw job
PR #72482107883389313880064
PR #68072107880778506440704
PR #71432107875990943305728
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #74052107765914286952448
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
PR #73972107681070840287232
ERROR CODE: DeadlineExceeded; provider Microsoft.Network
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Network
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e1014.08%10 of 71 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 20 · Oct 5: 78 · Oct 6: 57 · Oct 7: 23none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (11)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.235
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.12
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (10)
Date (UTC)Associated PRProw job
PR #72482107883389313880064
PR #68072107880778506440704
PR #74142107879778638893056
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
alert912.68%9 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsOct 1: 6 · Oct 2: 5 · Oct 3: 2 · Oct 4: 0 · Oct 5: 3 · Oct 6: 6 · Oct 7: 9none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T21:50:06Z
  Ended:       2026-10-07T21:56:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3227776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta6nelbkbeh1s64mru6t7ohgsjall8a-h3w3a5j6l2a4p9s", pod="openshift-apiserver-d5448bdcd-ltz5s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ta6nelbkbeh1s64mru6t7ohgsjall8a-h3w3a5j6l2a4p9s/openshift-apiserver-d5448bdcd-ltz5s has been in a non-ready state for longer than 5 minutes.
alert fired 9 time(s) (2 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T20:35:02Z
  Ended:       2026-10-07T20:41:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7631744-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta5emb000bsfom310889hpe5casu1jr-v5k9k8q2w5x0q6v", pod="openshift-apiserver-b9fcdfdb5-55mlh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ta5emb000bsfom310889hpe5casu1jr-v5k9k8q2w5x0q6v/openshift-apiserver-b9fcdfdb5-55mlh has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-07T20:35:02Z
  Ended:       2026-10-07T20:41:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7631744-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta5fdlm2qo2eb32ju5uasrq631306b1-cilium-cluster", pod="multus-admission-controller-578b7846f7-h57x6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ta5fdlm2qo2eb32ju5uasrq631306b1-cilium-cluster/multus-admission-controller-578b7846f7-h57x6 has been in a non-ready state for longer than 5 minutes.
alert fired 6 time(s) (3 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T18:11:35Z
  Ended:       2026-10-07T18:21:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3880064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta3l68gqrcq7gs6igibvm68inmua3g6-u0t5l5t9k7x3r9l", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ta3l68gqrcq7gs6igibvm68inmua3g6-u0t5l5t9k7x3r9l/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-07T18:11:35Z
  Ended:       2026-10-07T18:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3880064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta3kutmkbpo2kifvv50cfrikfbop2vu-y5c5w0x7t8g5d0d", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ta3kutmkbpo2kifvv50cfrikfbop2vu-y5c5w0x7t8g5d0d/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-07T18:12:33Z
  Ended:       2026-10-07T18:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3880064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta3l4ibb8m6kaqmai8gv3qag7e9qrug-t0m8d4r3h1i2g1k", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ta3l4ibb8m6kaqmai8gv3qag7e9qrug-t0m8d4r3h1i2g1k/etcd-1 has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire9
Affected runs (9)
Date (UTC)Associated PRProw job
PR #63642107926490833227776
PR #73992107908786537631744
PR #72482107883389313880064
PR #74112107879778164936704
PR #74102107844183950626816
PR #63642107828301232869376
PR #73972107751278686769152
PR #74022107717187903950848
PR #74002107676117312737280
alert [svc] KustoLogsDataStale fired
full failure pattern:
alert [svc] KustoLogsDataStale fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert45.63%4 of 71 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsOct 1: 3 · Oct 2: 2 · Oct 3: 0 · Oct 4: 0 · Oct 5: 6 · Oct 6: 5 · Oct 7: 4none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T02:13:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5347456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.24:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347456/providers/microsoft.monitor/accounts/services-j5347456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8fb68546b-b5vcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5347456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T02:14:49Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5347456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.24:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347456/providers/microsoft.monitor/accounts/services-j5347456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8fb68546b-b5vcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5347456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T19:18:35Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j6440704-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.112:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6440704/providers/microsoft.monitor/accounts/services-j6440704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6f8d84d4d4-wkcql", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j6440704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T19:18:35Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j6440704-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.112:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6440704/providers/microsoft.monitor/accounts/services-j6440704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6f8d84d4d4-wkcql", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j6440704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:35:53Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9974528-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9974528/providers/microsoft.monitor/accounts/services-j9974528", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6c57d99f8b-ljtj9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9974528-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:35:53Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9974528-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9974528/providers/microsoft.monitor/accounts/services-j9974528", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6c57d99f8b-ljtj9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9974528-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #72882107979694195347456
PR #68072107880778506440704
PR #74072107818144989974528
PR #74002107676117312737280
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
alert22.82%2 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 1: 5 · Oct 2: 1 · Oct 3: 0 · Oct 4: 0 · Oct 5: 7 · Oct 6: 7 · Oct 7: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:34:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8893056-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.61:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8893056/providers/microsoft.monitor/accounts/hcps-j8893056", namespace="ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-n8krx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:34:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8893056-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.61:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8893056/providers/microsoft.monitor/accounts/hcps-j8893056", namespace="ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-n8krx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:29:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3305728-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3305728/providers/microsoft.monitor/accounts/hcps-j3305728", namespace="ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gmh4h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:29:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3305728-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3305728/providers/microsoft.monitor/accounts/hcps-j3305728", namespace="ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gmh4h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74142107879778638893056
PR #71432107875990943305728
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
alert22.82%2 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 1: 0 · Oct 2: 3 · Oct 3: 1 · Oct 4: 0 · Oct 5: 4 · Oct 6: 5 · Oct 7: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:02:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-c4d6b786-bgkk8", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4d6b786-bgkk8 has been in a non-ready state for longer than 5 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T02:42:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5628416-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-445sp", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-445sp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T02:43:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5628416-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-gpz7w", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-gpz7w has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73052107961335642853376
PR #73972107647281405628416
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
alert11.41%1 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 1: 5 · Oct 2: 1 · Oct 3: 0 · Oct 4: 0 · Oct 5: 2 · Oct 6: 5 · Oct 7: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:21:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8893056-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.61:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8893056/providers/microsoft.monitor/accounts/hcps-j8893056", namespace="ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics", pod="cloud-network-config-controller-8445d77f54-dbcgg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b63493d6-07ef-4acd-8041-60edf966639a"
  Description: Pod ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics/cloud-network-config-controller-8445d77f54-dbcgg (controller) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74142107879778638893056
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e11.41%1 of 71 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 1: 1 · Oct 2: 1 · Oct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 0 · Oct 7: 1PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x1e187b528690>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x1e187b528690>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #70972107680144293367808
alert [svc] MaestroAgentMetricsProxyDown fired
full failure pattern:
alert [svc] MaestroAgentMetricsProxyDown fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
alert11.41%1 of 71 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 2 · Oct 7: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T02:35:12Z
  Ended:       2026-10-07T02:42:11Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5628416-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972107647281405628416
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e1419.72%14 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 15PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x37d635eafd40>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lpkn85dzg6wg" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191",
      "name": "321127fe-bb12-4e32-b036-dd251357b191",
      "status": "Failed",
      "startTime": "2026-10-07T20:56:46.02075406Z",
      "endTime": "2026-10-07T21:25:46.806941683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x37d635eafd40>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lpkn85dzg6wg" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191",
      "name": "321127fe-bb12-4e32-b036-dd251357b191",
      "status": "Failed",
      "startTime": "2026-10-07T20:56:46.02075406Z",
      "endTime": "2026-10-07T21:25:46.806941683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2a2d2cf153c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qtdjcjq4g46d" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "name": "d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "status": "Failed",
      "startTime": "2026-10-07T18:35:04.887969322Z",
      "endTime": "2026-10-07T19:04:11.73701564Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2a2d2cf153c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qtdjcjq4g46d" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "name": "d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "status": "Failed",
      "startTime": "2026-10-07T18:35:04.887969322Z",
      "endTime": "2026-10-07T19:04:11.73701564Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xf6ccd9b14e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-b4g4j9tj9qn5" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "name": "6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "status": "Failed",
      "startTime": "2026-10-07T18:34:53.377476522Z",
      "endTime": "2026-10-07T19:03:57.309891257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xf6ccd9b14e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-b4g4j9tj9qn5" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "name": "6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "status": "Failed",
      "startTime": "2026-10-07T18:34:53.377476522Z",
      "endTime": "2026-10-07T19:03:57.309891257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool14
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (14)
Date (UTC)Associated PRProw job
PR #73252107916955439599616
PR #72482107883389313880064
PR #68072107880778506440704
PR #74142107879778638893056
PR #71432107875990943305728
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #74052107765914286952448
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
PR #73972107681070840287232
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
full failure pattern:
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e57.04%5 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-zmb9zlf6bqmq/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-zmb9zlf6bqmq/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-w8w2hprnrv4g/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-w8w2hprnrv4g/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-cbtpkz6skzqd/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-cbtpkz6skzqd/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #74152107928469479362560
PR #72882107904235558408192
PR #72272107836958540566528
PR #73982107769248779603968
PR #73102107739528360890368
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
full failure pattern:
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e45.63%4 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e8a40f9d110>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e8a41056380>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e8a40efe990>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e8a40f9d110>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e8a41056380>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e8a40efe990>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x21f6a70e8e10>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x21f6a69d2ce0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x21f6a85f6620>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x21f6a70e8e10>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x21f6a69d2ce0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x21f6a85f6620>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x2b3cb6331890>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x2b3cb6202180>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x2b3cb57e7010>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x2b3cb6331890>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x2b3cb6202180>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x2b3cb57e7010>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #73252107970662109810688
PR #73252107916955439599616
PR #74072107818144989974528
PR #74002107676117312737280
alert [svc] LeaderElectionLeaseStale fired
full failure pattern:
alert [svc] LeaderElectionLeaseStale fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
alert34.23%3 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 5 · Oct 6: 6 · Oct 7: 3none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T01:40:45Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1217408-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1217408/providers/microsoft.monitor/accounts/services-j1217408", namespace="klusterlet-2ta9fcfdh71dkv4knt12m1ug0ranvkol", pod="arohcp-monitor-kube-state-metrics-7988dc648c-58rr2", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta9fcfdh71dkv4knt12m1ug0ranvkol on cluster ci01-j1217408-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:39:35Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j6520192-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.219:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6520192/providers/microsoft.monitor/accounts/services-j6520192", namespace="klusterlet-2ta7n5c8id8chjml49ukhi22658fcgoj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-lkfg4", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta7n5c8id8chjml49ukhi22658fcgoj on cluster ci01-j6520192-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T21:31:10Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3612672-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3612672/providers/microsoft.monitor/accounts/services-j3612672", namespace="klusterlet-2ta5r5t5b117do0a5grfaj0nbn80qof7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zcpkv", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta5r5t5b117do0a5grfaj0nbn80qof7 on cluster ci01-j3612672-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T21:42:11Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3612672-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3612672/providers/microsoft.monitor/accounts/services-j3612672", namespace="klusterlet-2ta5r55d86kstjo2bp5n4d3bvb279pda", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zcpkv", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta5r55d86kstjo2bp5n4d3bvb279pda on cluster ci01-j3612672-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #73612107979674851217408
PR #74122107955248256520192
PR #74142107916465603612672
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e34.23%3 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-ff6jdg"
Unexpected error:
    <*fmt.wrapError | 0x17d44c09e520>: 
    DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x17d44c09e500>{
            msg: "DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-ff6jdg"
Unexpected error:
    <*fmt.wrapError | 0x17d44c09e520>: 
    DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x17d44c09e500>{
            msg: "DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-vdz5wq"
Unexpected error:
    <*fmt.wrapError | 0x68ffbb0c4e0>: 
    DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x68ffbb0c4c0>{
            msg: "DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-vdz5wq"
Unexpected error:
    <*fmt.wrapError | 0x68ffbb0c4e0>: 
    DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x68ffbb0c4c0>{
            msg: "DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x2a03aec70780>: 
    DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x2a03aec70740>{
            msg: "DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x2a03aec70780>: 
    DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x2a03aec70740>{
            msg: "DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (3)
Date (UTC)Associated PRProw job
PR #74172107967978476670976
PR #73102107739528360890368
PR #74022107717187903950848
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 135none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f8378088558>: 
    resource group "kusto-logs-zdcz6ngd7l8s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zdcz6ngd7l8s" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "name": "ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "status": "Failed",
      "startTime": "2026-10-07T13:55:49.637158017Z",
      "endTime": "2026-10-07T14:19:49.807578144Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vu3tto21eb5vs2oeuv4d035r4kh9q still exists (deletion dispatched at 2026-10-07T13:56:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f8378088558>: 
    resource group "kusto-logs-zdcz6ngd7l8s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zdcz6ngd7l8s" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "name": "ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "status": "Failed",
      "startTime": "2026-10-07T13:55:49.637158017Z",
      "endTime": "2026-10-07T14:19:49.807578144Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vu3tto21eb5vs2oeuv4d035r4kh9q still exists (deletion dispatched at 2026-10-07T13:56:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xf860122e060>: 
    resource group "rg-zstream-upgrade-4-21-d7rnngth6rmt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-s5wrnm" in resourcegroup="rg-zstream-upgrade-4-21-d7rnngth6rmt" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "name": "de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "status": "Failed",
      "startTime": "2026-10-07T14:14:15.375309734Z",
      "endTime": "2026-10-07T14:38:20.14126941Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtq0dj9uuqrc354smigdjo5gckvsa still exists (deletion dispatched at 2026-10-07T14:14:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xf860122e060>: 
    resource group "rg-zstream-upgrade-4-21-d7rnngth6rmt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-s5wrnm" in resourcegroup="rg-zstream-upgrade-4-21-d7rnngth6rmt" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "name": "de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "status": "Failed",
      "startTime": "2026-10-07T14:14:15.375309734Z",
      "endTime": "2026-10-07T14:38:20.14126941Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtq0dj9uuqrc354smigdjo5gckvsa still exists (deletion dispatched at 2026-10-07T14:14:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3d715b6dc0c0>: 
    resource group "admin-api-serialconsole-kgtb4656t7wx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-kgtb4656t7wx" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "name": "63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "status": "Failed",
      "startTime": "2026-10-07T14:05:26.627856798Z",
      "endTime": "2026-10-07T14:29:30.238184303Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtu5p75nk86f54m2uds6vreosd2p2 still exists (deletion dispatched at 2026-10-07T14:05:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3d715b6dc0c0>: 
    resource group "admin-api-serialconsole-kgtb4656t7wx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-kgtb4656t7wx" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "name": "63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "status": "Failed",
      "startTime": "2026-10-07T14:05:26.627856798Z",
      "endTime": "2026-10-07T14:29:30.238184303Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtu5p75nk86f54m2uds6vreosd2p2 still exists (deletion dispatched at 2026-10-07T14:05:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (68)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.232
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.12
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.21 minor to 4.22 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.232
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
PR #74002107676117312737280
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xf53ccb2c090>: 
    resource group "rg-cordon-test-zfbgp6td9bvg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-zfbgp6td9bvg" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "name": "3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "status": "Failed",
      "startTime": "2026-10-07T17:55:02.189604085Z",
      "endTime": "2026-10-07T18:19:08.046890686Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3aagkkq4l3ogcvf4h8o101v6q3m76 still exists (deletion dispatched at 2026-10-07T17:55:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xf53ccb2c090>: 
    resource group "rg-cordon-test-zfbgp6td9bvg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-zfbgp6td9bvg" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "name": "3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "status": "Failed",
      "startTime": "2026-10-07T17:55:02.189604085Z",
      "endTime": "2026-10-07T18:19:08.046890686Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3aagkkq4l3ogcvf4h8o101v6q3m76 still exists (deletion dispatched at 2026-10-07T17:55:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3e339eaf0eb8>: 
    resource group "autoscaling-cluster-wq59b44864g7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wq59b44864g7" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "name": "486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "status": "Failed",
      "startTime": "2026-10-07T17:54:04.264847355Z",
      "endTime": "2026-10-07T18:18:08.23090916Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta38okksp4b36cn28fmfj8oja6islo0 still exists (deletion dispatched at 2026-10-07T17:54:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3e339eaf0eb8>: 
    resource group "autoscaling-cluster-wq59b44864g7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wq59b44864g7" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "name": "486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "status": "Failed",
      "startTime": "2026-10-07T17:54:04.264847355Z",
      "endTime": "2026-10-07T18:18:08.23090916Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta38okksp4b36cn28fmfj8oja6islo0 still exists (deletion dispatched at 2026-10-07T17:54:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19e0b121c138>: 
    resource group "admin-api-serialconsole-8wkg2h2hv8wb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-8wkg2h2hv8wb" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "name": "52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "status": "Failed",
      "startTime": "2026-10-07T17:56:40.94108576Z",
      "endTime": "2026-10-07T18:20:42.13673234Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cm8fj69efgottp7couqt5o1bu2r2 still exists (deletion dispatched at 2026-10-07T17:56:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19e0b121c138>: 
    resource group "admin-api-serialconsole-8wkg2h2hv8wb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-8wkg2h2hv8wb" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "name": "52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "status": "Failed",
      "startTime": "2026-10-07T17:56:40.94108576Z",
      "endTime": "2026-10-07T18:20:42.13673234Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cm8fj69efgottp7couqt5o1bu2r2 still exists (deletion dispatched at 2026-10-07T17:56:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71432107875990943305728
PR #74002107875420446658560
Command failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-<id>/pipeline@sha256:<hex> arohcpsvcdev.azurecr.io/ci-op-<id>/pipeline:<hex> --from-registry-confi...
full failure pattern:
Command failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-<id>/pipeline@sha256:<hex> arohcpsvcdev.azurecr.io/ci-op-<id>/pipeline:<hex> --from-registry-config /tmp/tmp.<id>/containers/auth.json --to-registry-config
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
provision22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
time=2026-10-07T02:30:15.367Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> fleet.image.registry <nil>}:{<nil> fleet.image.repository <nil>}@{<nil> fleet.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-07T02:32:48.216Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.lx56KPpMDi/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-jhx30i3q/pipeline@sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 to arohcpsvcdev.azurecr.io/ci-op-jhx30i3q/pipeline:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604.\nThe image will still be available under it's original digest sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 in the target registry.\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unavailable: service unavailable: health check failed: please see /debug/health\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unavailable: service unavailable: health check failed: please see /debug/health\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-jhx30i3q/pipeline@sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 arohcpsvcdev.azurecr.io/ci-op-jhx30i3q/pipeline:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 --from-registry-config /tmp/tmp.lx56KPpMDi/containers/auth.json --to-registry-config /tmp/tmp.lx56KPpMDi/containers/auth.json\n exit status 1"
time=2026-10-07T02:34:43.899Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> kubeApplier.image.registry <nil>}:{<nil> kubeApplier.image.repository <nil>}@{<nil> kubeApplier.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-07T02:35:51.286Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.BaWVNiNbL6/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-n30v64rp/pipeline@sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 to arohcpsvcdev.azurecr.io/ci-op-n30v64rp/pipeline:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894.\nThe image will still be available under it's original digest sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 in the target registry.\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-n30v64rp/pipeline@sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 arohcpsvcdev.azurecr.io/ci-op-n30v64rp/pipeline:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 --from-registry-config /tmp/tmp.BaWVNiNbL6/containers/auth.json --to-registry-config /tmp/tmp.BaWVNiNbL6/containers/auth.json\n exit status 1"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Fleet/global/mirror-image1
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.KubeApplier/global/mirror-image1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71052107647565049630720
PR #73972107647281405628416
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x213e82b661e0>: 
    resource group "rg-cluster-back-version-s9llrx7qtffw": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-s9llrx7qtffw", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x213e826ac820>{
                msg: "resource group \"rg-cluster-back-version-s9llrx7qtffw\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x213e826ac800>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x213e83e111d0>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x213e82b661e0>: 
    resource group "rg-cluster-back-version-s9llrx7qtffw": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-s9llrx7qtffw", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x213e826ac820>{
                msg: "resource group \"rg-cluster-back-version-s9llrx7qtffw\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x213e826ac800>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x213e83e111d0>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c28962ca078>: 
    resource group "rg-cluster-back-version-kcpxjfv4fpw8": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c289468e340>{
                msg: "resource group \"rg-cluster-back-version-kcpxjfv4fpw8\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x3c289468e320>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x3c2894a68660>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c28962ca078>: 
    resource group "rg-cluster-back-version-kcpxjfv4fpw8": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c289468e340>{
                msg: "resource group \"rg-cluster-back-version-kcpxjfv4fpw8\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x3c289468e320>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x3c2894a68660>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
PR #74002107676117312737280
Process did not finish before 15m0s timeout
full failure pattern:
Process did not finish before 15m0s timeout
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
unknown22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-10-07T15:54:40Z"}
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-10-07T06:30:56Z"}
Contributing tests (1)
Failed AtJobTestSupport
unknownpull-ci-Azure-ARO-HCP-main-e2e-parallelJob run should complete before timeout2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
PR #74002107676117312737280
alert [hcp] HCPClusterOperatorUnavailable fired
full failure pattern:
alert [hcp] HCPClusterOperatorUnavailable fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
alert22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T14:28:55Z
  Ended:       2026-10-07T15:55:54Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtqjq0sahnttfr0obenao468e4qlb-e0e2j3m7x8c7o1b", region="westus3", severity="warning"
  Description: 10 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtqjq0sahnttfr0obenao468e4qlb-e0e2j3m7x8c7o1b (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:52Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vuf7r0mkidc4qe1opuvsfmq0jlia8-p9j3q3c5l3l9j9t", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vuf7r0mkidc4qe1opuvsfmq0jlia8-p9j3q3c5l3l9j9t (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:52Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vu5443vh0o9q88a97n0snmv6qq2pb-acr-pull", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vu5443vh0o9q88a97n0snmv6qq2pb-acr-pull (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:52Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vu0gdmpn5k5d5bgvk4606cnb2ql44-s5l1e0o2a6j6m8n", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vu0gdmpn5k5d5bgvk4606cnb2ql44-s5l1e0o2a6j6m8n (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 5:
  State:       Resolved
  Started:     2026-10-07T15:27:53Z
  Ended:       2026-10-07T15:46:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtnemv5o1grorom2pjgo9il7l5cgq-l7p8p3x8g9o9l5c", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtnemv5o1grorom2pjgo9il7l5cgq-l7p8p3x8g9o9l5c (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtt5ll44f8b35n9lm1o4n0oims33s-private-kas", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtt5ll44f8b35n9lm1o4n0oims33s-private-kas (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 7:
  State:       Resolved
  Started:     2026-10-07T15:27:53Z
  Ended:       2026-10-07T15:47:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtu5p75nk86f54m2uds6vreosd2p2-v0n4p7w8h2f0o9a", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtu5p75nk86f54m2uds6vreosd2p2-v0n4p7w8h2f0o9a (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 8:
  State:       Resolved
  Started:     2026-10-07T15:27:53Z
  Ended:       2026-10-07T15:47:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vu3ea2tjsap1sokp49bjsntgc3llb-u9p1t5k3r2x5x0n", region="westus3", severity="warning"
  Description: 8 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vu3ea2tjsap1sokp49bjsntgc3llb-u9p1t5k3r2x5x0n (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T05:10:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlcah4jgmcp45v4eq4gnjhrn1t86e-ingress-cert-le", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlcah4jgmcp45v4eq4gnjhrn1t86e-ingress-cert-le (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 2:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:30:07Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlr9uea4a002u1vobs1eamhd9ij0g-l5p7g4y6r1u6h3x", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlr9uea4a002u1vobs1eamhd9ij0g-l5p7g4y6r1u6h3x (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 3:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:29:08Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nldv9i221aq0hikhv2urfttr6p4ag-j3v3v2v9p0l8t7b", region="westus3", severity="warning"
  Description: 8 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nldv9i221aq0hikhv2urfttr6p4ag-j3v3v2v9p0l8t7b (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-07T06:04:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlhakelm48kkcoo98ifgh0s48qjr9-x7k8b4r2l9a9q2i", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlhakelm48kkcoo98ifgh0s48qjr9-x7k8b4r2l9a9q2i (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 5:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:29:08Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nleqkohnk6u75anrikd0ihfs6jmk6-i8v1g1d3d9x0t2s", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nleqkohnk6u75anrikd0ihfs6jmk6-i8v1g1d3d9x0t2s (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 6:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:32:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nllqoph4r7lp0chrtmvsdfajj63mb-s0d9l7t2s9f2t9s", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nllqoph4r7lp0chrtmvsdfajj63mb-s0d9l7t2s9f2t9s (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 7:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:30:07Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nldc2p59tvtq108i7fn6ktamnpfd9-private-kas", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nldc2p59tvtq108i7fn6ktamnpfd9-private-kas (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 8:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:33:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlp25sdnfj4rsjsnnajlljlta2m49-acr-pull", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlp25sdnfj4rsjsnnajlljlta2m49-acr-pull (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
PR #74002107676117312737280
alert [svc] BackendControllerRetryHotLoop fired
full failure pattern:
alert [svc] BackendControllerRetryHotLoop fired
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
alert22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 1 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T14:03:14Z
  Ended:       2026-10-07T14:20:15Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j9974528-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T04:40:19Z
  Ended:       2026-10-07T04:55:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j2737280-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
PR #74002107676117312737280
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
full failure pattern:
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e22.82%2 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x50103f9c9c0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-d5s257gp6qv6" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-d5s257gp6qv6\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x50103f9c9c0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-d5s257gp6qv6" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-d5s257gp6qv6\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x2c581ef3e4a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wcbt6xkttfqm" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-wcbt6xkttfqm\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x2c581ef3e4a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wcbt6xkttfqm" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-wcbt6xkttfqm\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69542107841500657225728
PR #63642107828301232869376
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x9d4c3cd8090>: 
    resource group "admin-api-breakglass-6smbnkdj4hzs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-6smbnkdj4hzs" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "name": "1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "status": "Failed",
      "startTime": "2026-10-07T17:53:56.233010929Z",
      "endTime": "2026-10-07T18:18:01.34812688Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cjpfhsndlhr2qta8lp3hig1f25is still exists (deletion dispatched at 2026-10-07T17:53:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x9d4c3cd8090>: 
    resource group "admin-api-breakglass-6smbnkdj4hzs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-6smbnkdj4hzs" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "name": "1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "status": "Failed",
      "startTime": "2026-10-07T17:53:56.233010929Z",
      "endTime": "2026-10-07T18:18:01.34812688Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cjpfhsndlhr2qta8lp3hig1f25is still exists (deletion dispatched at 2026-10-07T17:53:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25222e7880f0>: 
    resource group "rg-zstream-upgrade-4-20-42mlb9pj8467": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-2ns5vg" in resourcegroup="rg-zstream-upgrade-4-20-42mlb9pj8467" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d",
      "name": "bb6159e1-d013-447b-ad27-c209bc94526d",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:52.899645021Z",
      "endTime": "2026-10-07T18:25:01.15966395Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebl56c239c6lvk3po879sfe0nus still exists (deletion dispatched at 2026-10-07T18:00:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25222e7880f0>: 
    resource group "rg-zstream-upgrade-4-20-42mlb9pj8467": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-2ns5vg" in resourcegroup="rg-zstream-upgrade-4-20-42mlb9pj8467" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d",
      "name": "bb6159e1-d013-447b-ad27-c209bc94526d",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:52.899645021Z",
      "endTime": "2026-10-07T18:25:01.15966395Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebl56c239c6lvk3po879sfe0nus still exists (deletion dispatched at 2026-10-07T18:00:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27dd6af64318>: 
    resource group "ingress-cert-le-h6vkqmnzj7lf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-h6vkqmnzj7lf" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "name": "6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "status": "Failed",
      "startTime": "2026-10-07T18:01:35.349390537Z",
      "endTime": "2026-10-07T18:25:41.362314296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebs5l8pbehlv64f21cvh9tk83jo still exists (deletion dispatched at 2026-10-07T18:01:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27dd6af64318>: 
    resource group "ingress-cert-le-h6vkqmnzj7lf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-h6vkqmnzj7lf" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "name": "6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "status": "Failed",
      "startTime": "2026-10-07T18:01:35.349390537Z",
      "endTime": "2026-10-07T18:25:41.362314296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebs5l8pbehlv64f21cvh9tk83jo still exists (deletion dispatched at 2026-10-07T18:01:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74002107875420446658560
alert [svc] ServiceMemoryDrift fired
full failure pattern:
alert [svc] ServiceMemoryDrift fired
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
alert11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:26:41Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9974528-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jrr5z", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jrr5z (namespace arobit) on cluster ci01-j9974528-svc has averaged 124.3% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
alert11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
alert fired 9 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:27:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:27:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-07T23:34:27Z
  Ended:       2026-10-07T23:43:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta829bp1fhgrfupabf34hjhbao8sjia", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta829bp1fhgrfupabf34hjhbao8sjia/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-07T23:35:31Z
  Ended:       2026-10-07T23:45:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta82bca3t1j71tbsbphnabaj9385rs2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta82bca3t1j71tbsbphnabaj9385rs2/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-07T23:35:32Z
  Ended:       2026-10-07T23:54:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-07T23:59:27Z
  Ended:       2026-10-08T00:08:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8e00ategjs6jdt7dm9iouhjbvp29q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8e00ategjs6jdt7dm9iouhjbvp29q/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-08T00:00:27Z
  Ended:       2026-10-08T00:10:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8erefr60nd4djngqkit7pco8mj14u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8erefr60nd4djngqkit7pco8mj14u/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-08T00:00:27Z
  Ended:       2026-10-08T00:07:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8f7d8ssfok4nfel8v6oomstor9osq", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8f7d8ssfok4nfel8v6oomstor9osq/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-08T00:02:28Z
  Ended:       2026-10-08T00:14:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8fb8j9edjipdiide2u4g57sv913hs", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8fb8j9edjipdiide2u4g57sv913hs/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73052107961335642853376
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x219764076460>: 
    DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x219764076440>{
            msg: "DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x219764076460>: 
    DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x219764076440>{
            msg: "DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972107751278686769152
alert [svc] KubeDeploymentRolloutStuck fired
full failure pattern:
alert [svc] KubeDeploymentRolloutStuck fired
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
alert11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:37:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-07T23:45:38Z
  Ended:       2026-10-07T23:54:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T00:09:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73052107961335642853376
ERROR: Please run 'az login' to setup account.
full failure pattern:
ERROR: Please run 'az login' to setup account.
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
provision11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
time=2026-10-07T09:04:20.332Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 description="Step swift-vnet\n  Kind: Shell\n  Command: ./swift-vnet.sh\n"
time=2026-10-07T09:04:22.138Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: Launching container group swift-vnet-aks-net as globalMSI...\nERROR: Please run 'az login' to setup account.\n exit status 1"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/swift-vnet (stamp=1)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73982107750698572582912
failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1 context deadline exceeded
full failure pattern:
failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1 context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
provision11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
time=2026-10-07T19:22:31.326Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events description="Step kube-events\n Kind: Helm\n"
time=2026-10-07T19:22:32.066Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events values="map[enabled:true image:map[digest:sha256:7c0c84fdd1dde7d32754d602c60e7432108ecef850d5bcd2efc549110fbda458 registry:kubernetesshared.azurecr.io repository:shared/kube-events] resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]]]"
time=2026-10-07T19:22:32.066Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-10-07T19:22:32.374Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events namespace=monitoring
time=2026-10-07T19:22:32.684Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events dryRun=false
time=2026-10-07T19:22:32.684Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-10-07T19:22:32.822Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-10-07T19:27:38.910Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-07T19:27:38.911Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/kube-events1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73612107904302923124736
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 2 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x1f1b762f1400>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-dzl9ch5fblqj" to finish creating: GET https://rp.j2675200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "name": "597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "status": "Failed",
      "startTime": "2026-10-07T17:41:10.05474314Z",
      "endTime": "2026-10-07T18:10:19.11453378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x1f1b762f1400>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-dzl9ch5fblqj" to finish creating: GET https://rp.j2675200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "name": "597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "status": "Failed",
      "startTime": "2026-10-07T17:41:10.05474314Z",
      "endTime": "2026-10-07T18:10:19.11453378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73102107872947002675200
ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershi...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; hypershift NodePool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; hypershift NodePool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc0b67484b00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-b6bhhgvdr2ld" to finish updating: GET https://rp.j0967424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "name": "9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "status": "Failed",
      "startTime": "2026-10-07T18:15:51.4879876Z",
      "endTime": "2026-10-07T18:34:57.661146093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is 2, want 3; hypershift NodePool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc0b67484b00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-b6bhhgvdr2ld" to finish updating: GET https://rp.j0967424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "name": "9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "status": "Failed",
      "startTime": "2026-10-07T18:15:51.4879876Z",
      "endTime": "2026-10-07T18:34:57.661146093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is 2, want 3; hypershift NodePool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73222107875761300967424
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3ed0c46420d8>: 
    resource group "rg-negative-tests-z8rjxzl5wzq5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8rjxzl5wzq5" to finish deleting: GET https://rp.j8408192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "name": "92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "status": "Failed",
      "startTime": "2026-10-07T20:31:10.664805555Z",
      "endTime": "2026-10-07T20:55:12.515323239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3ed0c46420d8>: 
    resource group "rg-negative-tests-z8rjxzl5wzq5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8rjxzl5wzq5" to finish deleting: GET https://rp.j8408192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "name": "92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "status": "Failed",
      "startTime": "2026-10-07T20:31:10.664805555Z",
      "endTime": "2026-10-07T20:55:12.515323239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72882107904235558408192
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no mat...
full failure pattern:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x1d103f80a620>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
      - pod pull-secret-test container pull-test: ContainerCreating ()
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        err: <*errors.errorString | 0x1d103f3186b0>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x1d103f80a620>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
      - pod pull-secret-test container pull-test: ContainerCreating ()
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        err: <*errors.errorString | 0x1d103f3186b0>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73252107817608098091008
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...
full failure pattern:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T20:13:47
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0x4aa5231ef20>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);
    let _endTime = datetime(2026-10-07T20:13:47.1720653Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-gg7xds6svpzk'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T20:13:54.1282773Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77", ActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", SubActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityStack:"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x4aa53d27068>{
            errs: [
                <*fmt.wrapError | 0x4aa5231ef00>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x4aa5231eec0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x4aa52207ce0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x4aa52207ca0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x4aa52820300>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0x4aa5231ef20>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);
    let _endTime = datetime(2026-10-07T20:13:47.1720653Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-gg7xds6svpzk'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T20:13:54.1282773Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77", ActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", SubActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityStack:"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x4aa53d27068>{
            errs: [
                <*fmt.wrapError | 0x4aa5231ef00>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x4aa5231eec0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x4aa52207ce0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x4aa52207ca0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x4aa52820300>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72882107904235558408192
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.41%1 of 71 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x1df0828207f8>: 
    resource group "rg-zstream-upgrade-4-23-4xq25ksc85ts": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-23-vhm2c5" in resourcegroup="rg-zstream-upgrade-4-23-4xq25ksc85ts" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7f203186-8575-4f16-9fec-af4db81dcdaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7f203186-8575-4f16-9fec-af4db81dcdaf",
      "name": "7f203186-8575-4f16-9fec-af4db81dcdaf",
      "status": "Failed",
      "startTime": "2026-10-07T18:02:18.08747435Z",
      "endTime": "2026-10-07T18:26:27.625877529Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x1df0828207f8>: 
    resource group "rg-zstream-upgrade-4-23-4xq25ksc85ts": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-23-vhm2c5" in resourcegroup="rg-zstream-upgrade-4-23-4xq25ksc85ts" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7f203186-8575-4f16-9fec-af4db81dcdaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7f203186-8575-4f16-9fec-af4db81dcdaf",
      "name": "7f203186-8575-4f16-9fec-af4db81dcdaf",
      "status": "Failed",
      "startTime": "2026-10-07T18:02:18.08747435Z",
      "endTime": "2026-10-07T18:26:27.625877529Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71432107875990943305728

Environment: INT

Failure patterns: 0 · Failures matched: 0 · Total runs: 4

No semantic signatures had matching failures in this window.

Environment: PROD

Failure patterns: 14 · Failures matched: 153 · Total runs: 7 · Runs affected: 6

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
e2e114.29%1 of 7 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsOct 1: 2 · Oct 2: 2 · Oct 3: 2 · Oct 4: 2 · Oct 5: 3 · Oct 6: 3 · Oct 7: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf"
Unexpected error:
    <*fmt.wrapError | 0x312a48ea2980>: 
    failed to create NodePool np-zgtwtf: failed starting nodepool creation "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf" in resourcegroup="rg-nightly-4-20-j42q4dvn7t74": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-j42q4dvn7t74/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zgtwtf/nodePools/np-zgtwtf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf"
Unexpected error:
    <*fmt.wrapError | 0x312a48ea2980>: 
    failed to create NodePool np-zgtwtf: failed starting nodepool creation "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf" in resourcegroup="rg-nightly-4-20-j42q4dvn7t74": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-j42q4dvn7t74/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zgtwtf/nodePools/np-zgtwtf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x32c6fec798e0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-vbjww8" in resourcegroup="rg-np-skip-minor-vbjww8-9dlhkbhpd9hb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vbjww8-9dlhkbhpd9hb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vbjww8/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x32c6fec798e0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-vbjww8" in resourcegroup="rg-np-skip-minor-vbjww8-9dlhkbhpd9hb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vbjww8-9dlhkbhpd9hb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vbjww8/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1829881ca3e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7pmxvz" in resourcegroup="rg-np-version-upgrade-7pmxvz-jjjscx2plvvj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pmxvz-jjjscx2plvvj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pmxvz/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1829881ca3e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7pmxvz" in resourcegroup="rg-np-version-upgrade-7pmxvz-jjjscx2plvvj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pmxvz-jjjscx2plvvj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pmxvz/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107969374202630144
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
e2e114.29%1 of 7 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsOct 1: 5 · Oct 2: 2 · Oct 3: 2 · Oct 4: 2 · Oct 5: 2 · Oct 6: 2 · Oct 7: 2STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x26d333bbc300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-ckbnsnb4ttgr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x26d333bbc300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-ckbnsnb4ttgr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x6c1c704d590>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6dmndbpnqjgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x6c1c704d590>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6dmndbpnqjgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107969374202630144
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e571.43%5 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 1: 6 · Oct 2: 5 · Oct 3: 1 · Oct 4: 12 · Oct 5: 1 · Oct 6: 6 · Oct 7: 43DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m"
Unexpected error:
    <*fmt.wrapError | 0x2ac110093d00>: 
    failed to create NodePool np-gcl76m: failed waiting for nodepool="np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m" in resourcegroup="rg-candidate-5-0-fvphxw8q5b7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "name": "891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "status": "Failed",
      "startTime": "2026-10-08T00:42:35.095507926Z",
      "endTime": "2026-10-08T01:11:35.931061942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m"
Unexpected error:
    <*fmt.wrapError | 0x2ac110093d00>: 
    failed to create NodePool np-gcl76m: failed waiting for nodepool="np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m" in resourcegroup="rg-candidate-5-0-fvphxw8q5b7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "name": "891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "status": "Failed",
      "startTime": "2026-10-08T00:42:35.095507926Z",
      "endTime": "2026-10-08T01:11:35.931061942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3133a1380440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-dfm6bjrfc7jr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "name": "30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:37.937984296Z",
      "endTime": "2026-10-08T00:36:41.067847774Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-nb26h-jsv7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-nb26h-jsv7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3133a1380440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-dfm6bjrfc7jr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "name": "30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:37.937984296Z",
      "endTime": "2026-10-08T00:36:41.067847774Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-nb26h-jsv7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-nb26h-jsv7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh"
Unexpected error:
    <*fmt.wrapError | 0x36df90d3b540>: 
    failed to create NodePool np-tlv9nh: failed waiting for nodepool="np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh" in resourcegroup="rg-candidate-4-22-k8bbp9vgvcln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e",
      "name": "5b294c96-939a-4202-a8fa-48f02c92df7e",
      "status": "Failed",
      "startTime": "2026-10-08T00:39:43.973177646Z",
      "endTime": "2026-10-08T01:08:45.917568185Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh"
Unexpected error:
    <*fmt.wrapError | 0x36df90d3b540>: 
    failed to create NodePool np-tlv9nh: failed waiting for nodepool="np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh" in resourcegroup="rg-candidate-4-22-k8bbp9vgvcln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e",
      "name": "5b294c96-939a-4202-a8fa-48f02c92df7e",
      "status": "Failed",
      "startTime": "2026-10-08T00:39:43.973177646Z",
      "endTime": "2026-10-08T01:08:45.917568185Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (26)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.213
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.223
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.233
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.13
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2107977652647235584
n/a2107969374202630144
n/a2107917271417491456
n/a2107872150416265216
n/a2107652362557460480
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e571.43%5 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 5DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x32955c70b520>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f92csqngd5f6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "name": "c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "status": "Failed",
      "startTime": "2026-10-08T01:32:50.695442989Z",
      "endTime": "2026-10-08T02:01:50.773194309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x32955c70b520>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f92csqngd5f6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "name": "c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "status": "Failed",
      "startTime": "2026-10-08T01:32:50.695442989Z",
      "endTime": "2026-10-08T02:01:50.773194309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1c11c2b2b440>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-npbf7r72xkcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "name": "26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "status": "Failed",
      "startTime": "2026-10-08T00:01:08.520978007Z",
      "endTime": "2026-10-08T00:30:17.207038187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1c11c2b2b440>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-npbf7r72xkcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "name": "26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "status": "Failed",
      "startTime": "2026-10-08T00:01:08.520978007Z",
      "endTime": "2026-10-08T00:30:17.207038187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d0dde8c3780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cdl8czz92fjt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8",
      "name": "1b34011e-02fa-407e-bf8e-697c866145e8",
      "status": "Failed",
      "startTime": "2026-10-07T21:30:13.060183446Z",
      "endTime": "2026-10-07T21:59:13.201730463Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d0dde8c3780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cdl8czz92fjt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8",
      "name": "1b34011e-02fa-407e-bf8e-697c866145e8",
      "status": "Failed",
      "startTime": "2026-10-07T21:30:13.060183446Z",
      "endTime": "2026-10-07T21:59:13.201730463Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2107977652647235584
n/a2107969374202630144
n/a2107917271417491456
n/a2107872150416265216
n/a2107652362557460480
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e457.14%4 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsOct 1: 0 · Oct 2: 1 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 1 · Oct 7: 5DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x30a375627b20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x30a375627ae0>{
            msg: "failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x30a375627a80>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x30a375627b20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x30a375627ae0>{
            msg: "failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x30a375627a80>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2c512700e018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bsqjlrm7bfgr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "name": "597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:05.991580079Z",
      "endTime": "2026-10-08T00:36:11.101154752Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2c512700e018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bsqjlrm7bfgr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "name": "597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:05.991580079Z",
      "endTime": "2026-10-08T00:36:11.101154752Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x42d11f950c8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2vrv7v7vlpcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d",
      "name": "9a289f01-1747-41f4-af5e-032e7e217e1d",
      "status": "Failed",
      "startTime": "2026-10-07T20:08:33.943284812Z",
      "endTime": "2026-10-07T20:37:40.405868199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x42d11f950c8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2vrv7v7vlpcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d",
      "name": "9a289f01-1747-41f4-af5e-032e7e217e1d",
      "status": "Failed",
      "startTime": "2026-10-07T20:08:33.943284812Z",
      "endTime": "2026-10-07T20:37:40.405868199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2107977652647235584
n/a2107917271417491456
n/a2107872150416265216
n/a2107652362557460480
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenSh...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
e2e228.57%2 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsOct 1: 0 · Oct 2: 6 · Oct 3: 15 · Oct 4: 16 · Oct 5: 17 · Oct 6: 18 · Oct 7: 17none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-p2kwgg" to 4.21.0-0.nightly-multi-2026-10-02-115050
Unexpected error:
    <*exported.ResponseError | 0xd94e927e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-xqwwvdsh9nb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-p2kwgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-p2kwgg" to 4.21.0-0.nightly-multi-2026-10-02-115050
Unexpected error:
    <*exported.ResponseError | 0xd94e927e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-xqwwvdsh9nb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-p2kwgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-dl424r" to 4.23.0-0.nightly-multi-2026-10-06-030402
Unexpected error:
    <*exported.ResponseError | 0x23e2acc0cbd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-kjs4dvf659k9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-dl424r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-dl424r" to 4.23.0-0.nightly-multi-2026-10-06-030402
Unexpected error:
    <*exported.ResponseError | 0x23e2acc0cbd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-kjs4dvf659k9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-dl424r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2661dd4d00c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-z8pk7tmfp9jj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2661dd4d00c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-z8pk7tmfp9jj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (17)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2107969374202630144
n/a2107652362557460480
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenS...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
e2e228.57%2 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsOct 1: 0 · Oct 2: 2 · Oct 3: 11 · Oct 4: 11 · Oct 5: 12 · Oct 6: 12 · Oct 7: 12none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1fa024921260>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-8hhqrj" resourcegroup="rg-np-dg-minor-8hhqrj-7f8bnzmkm895": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-8hhqrj-7f8bnzmkm895/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-8hhqrj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1fa024921260>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-8hhqrj" resourcegroup="rg-np-dg-minor-8hhqrj-7f8bnzmkm895": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-8hhqrj-7f8bnzmkm895/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-8hhqrj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x2e46f700980>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tdmf8r24g52m": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-tdmf8r24g52m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x2e46f700980>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tdmf8r24g52m": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-tdmf8r24g52m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x35088dfe27a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-dwmb74mwp62t": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-dwmb74mwp62t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x35088dfe27a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-dwmb74mwp62t": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-dwmb74mwp62t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (12)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2107969374202630144
n/a2107652362557460480
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube con...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 51none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z9xhpt9dxztr/cluster-candidate-4-22-8frl8d should provision
Unexpected error:
    <*fmt.wrapError | 0xc3c4040eec0>: 
    failed to create HCP cluster cluster-candidate-4-22-8frl8d: failed waiting for cluster="cluster-candidate-4-22-8frl8d" in resourcegroup="rg-candidate-4-22-z9xhpt9dxztr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "name": "6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.12350471Z",
      "endTime": "2026-10-07T11:13:04.942313802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z9xhpt9dxztr/cluster-candidate-4-22-8frl8d should provision
Unexpected error:
    <*fmt.wrapError | 0xc3c4040eec0>: 
    failed to create HCP cluster cluster-candidate-4-22-8frl8d: failed waiting for cluster="cluster-candidate-4-22-8frl8d" in resourcegroup="rg-candidate-4-22-z9xhpt9dxztr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "name": "6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.12350471Z",
      "endTime": "2026-10-07T11:13:04.942313802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2851e862a620>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ws854cfjbfc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "name": "5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:27.70826866Z",
      "endTime": "2026-10-07T12:10:34.84968826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2851e862a620>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ws854cfjbfc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "name": "5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:27.70826866Z",
      "endTime": "2026-10-07T12:10:34.84968826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x231265410c40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-m6pl2gptcdfp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "name": "3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:27.398564047Z",
      "endTime": "2026-10-07T12:08:34.791319585Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x231265410c40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-m6pl2gptcdfp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "name": "3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:27.398564047Z",
      "endTime": "2026-10-07T12:08:34.791319585Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (51)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x186c1f59fe00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-xkrgtz49hrh4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xkrgtz49hrh4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x186c1f59fe00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-xkrgtz49hrh4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xkrgtz49hrh4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x5e92b4ffb60>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-jsrm64gc6pjs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-jsrm64gc6pjs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x5e92b4ffb60>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-jsrm64gc6pjs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-jsrm64gc6pjs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0x1d4613307440>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-xccktj7qbpzs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0x1d4613307440>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-xccktj7qbpzs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpe...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsOct 1: 0 · Oct 2: 5 · Oct 3: 3 · Oct 4: 2 · Oct 5: 3 · Oct 6: 2 · Oct 7: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-gcjnnr" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x2862c3242150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-97btfzzdk5qt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gcjnnr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-gcjnnr" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x2862c3242150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-97btfzzdk5qt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gcjnnr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-wfqdx6" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x35670f110a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-m89bbfgxxvxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-wfqdx6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-wfqdx6" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x35670f110a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-m89bbfgxxvxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-wfqdx6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-p4lgwf" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x1b0947b50a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-2gpxkvcjbvk8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-p4lgwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-p4lgwf" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x1b0947b50a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-2gpxkvcjbvk8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-p4lgwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107652362557460480
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
full failure pattern:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsOct 1: 0 · Oct 2: 1 · Oct 3: 1 · Oct 4: 0 · Oct 5: 1 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x2e5cf51ae018>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2e5cf5120ca0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf5120c80>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4986960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2e5cf52a2040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf52a2020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4b086e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x2e5cf51ae018>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2e5cf5120ca0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf5120c80>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4986960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2e5cf52a2040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf52a2020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4b086e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107969374202630144
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x2fbb237581e0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kc94rh5nhxb8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kc94rh5nhxb8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2fbb237ab310>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x2fbb237581e0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kc94rh5nhxb8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kc94rh5nhxb8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2fbb237ab310>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e114.29%1 of 7 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x99270993880>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x99270993880>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107977652647235584

Environment: STG

Failure patterns: 6 · Failures matched: 60 · Total runs: 6 · Runs affected: 5

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e583.33%5 of 6 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsOct 1: 6 · Oct 2: 5 · Oct 3: 2 · Oct 4: 1 · Oct 5: 1 · Oct 6: 18 · Oct 7: 40DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xkh7k5" for cluster "cluster-nightly-4-22-xkh7k5"
Unexpected error:
    <*fmt.wrapError | 0x1e843b97f120>: 
    failed to create NodePool np-xkh7k5: failed waiting for nodepool="np-xkh7k5" for cluster "cluster-nightly-4-22-xkh7k5" in resourcegroup="rg-nightly-4-22-lnvbhdz2bflf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc30ffaf-307f-4841-b4a2-6d2c8334bf3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc30ffaf-307f-4841-b4a2-6d2c8334bf3b",
      "name": "dc30ffaf-307f-4841-b4a2-6d2c8334bf3b",
      "status": "Failed",
      "startTime": "2026-10-07T23:40:08.180826205Z",
      "endTime": "2026-10-08T00:09:12.758937229Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc to exist\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc to exist\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xkh7k5" for cluster "cluster-nightly-4-22-xkh7k5"
Unexpected error:
    <*fmt.wrapError | 0x1e843b97f120>: 
    failed to create NodePool np-xkh7k5: failed waiting for nodepool="np-xkh7k5" for cluster "cluster-nightly-4-22-xkh7k5" in resourcegroup="rg-nightly-4-22-lnvbhdz2bflf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc30ffaf-307f-4841-b4a2-6d2c8334bf3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc30ffaf-307f-4841-b4a2-6d2c8334bf3b",
      "name": "dc30ffaf-307f-4841-b4a2-6d2c8334bf3b",
      "status": "Failed",
      "startTime": "2026-10-07T23:40:08.180826205Z",
      "endTime": "2026-10-08T00:09:12.758937229Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc to exist\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-8jjmc to exist\nMachine a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-lnvbhdz2bflf-nightly-xkh7k5--managed/providers/Microsoft.Compute/virtualMachines/a2v2x5b8f7z8j6l-np-xkh7k5-rx767-sm68r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1a93c5372000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-v2qsz6wkc724" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93d34adc-e5de-4e29-8488-7bce9a8a626f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93d34adc-e5de-4e29-8488-7bce9a8a626f",
      "name": "93d34adc-e5de-4e29-8488-7bce9a8a626f",
      "status": "Failed",
      "startTime": "2026-10-07T23:17:29.82535222Z",
      "endTime": "2026-10-07T23:46:38.16324507Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u2g6d8v8s1g6k8i-np-1-szw76-mp2gq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-mp2gq to exist\nMachine u2g6d8v8s1g6k8i-np-1-szw76-rhh25: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-rhh25 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u2g6d8v8s1g6k8i-np-1-szw76-mp2gq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-mp2gq to exist\nMachine u2g6d8v8s1g6k8i-np-1-szw76-rhh25: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-rhh25 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1a93c5372000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-v2qsz6wkc724" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93d34adc-e5de-4e29-8488-7bce9a8a626f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93d34adc-e5de-4e29-8488-7bce9a8a626f",
      "name": "93d34adc-e5de-4e29-8488-7bce9a8a626f",
      "status": "Failed",
      "startTime": "2026-10-07T23:17:29.82535222Z",
      "endTime": "2026-10-07T23:46:38.16324507Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u2g6d8v8s1g6k8i-np-1-szw76-mp2gq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-mp2gq to exist\nMachine u2g6d8v8s1g6k8i-np-1-szw76-rhh25: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-rhh25 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u2g6d8v8s1g6k8i-np-1-szw76-mp2gq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-mp2gq to exist\nMachine u2g6d8v8s1g6k8i-np-1-szw76-rhh25: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-v2qsz6wkc724--managed/providers/Microsoft.Compute/virtualMachines/u2g6d8v8s1g6k8i-np-1-szw76-rhh25 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dhdr4d" for cluster "cluster-nightly-5-0-dhdr4d"
Unexpected error:
    <*fmt.wrapError | 0x1bb2d5ca8760>: 
    failed to create NodePool np-dhdr4d: failed waiting for nodepool="np-dhdr4d" for cluster "cluster-nightly-5-0-dhdr4d" in resourcegroup="rg-nightly-5-0-w2nf95ndpzrw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5845b729-520b-4e46-b237-ae83e6fffeab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5845b729-520b-4e46-b237-ae83e6fffeab",
      "name": "5845b729-520b-4e46-b237-ae83e6fffeab",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:07.401401662Z",
      "endTime": "2026-10-08T00:12:12.648695446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk to exist\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk to exist\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dhdr4d" for cluster "cluster-nightly-5-0-dhdr4d"
Unexpected error:
    <*fmt.wrapError | 0x1bb2d5ca8760>: 
    failed to create NodePool np-dhdr4d: failed waiting for nodepool="np-dhdr4d" for cluster "cluster-nightly-5-0-dhdr4d" in resourcegroup="rg-nightly-5-0-w2nf95ndpzrw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5845b729-520b-4e46-b237-ae83e6fffeab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5845b729-520b-4e46-b237-ae83e6fffeab",
      "name": "5845b729-520b-4e46-b237-ae83e6fffeab",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:07.401401662Z",
      "endTime": "2026-10-08T00:12:12.648695446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk to exist\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-n9bkk to exist\nMachine p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-w2nf95ndpzrw-nightly-dhdr4d--managed/providers/Microsoft.Compute/virtualMachines/p2s4e8y8s1w9m5p-np-dhdr4d-mg9mh-qzh8j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (24)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.213
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.223
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.233
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.13
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2107969374252961792
n/a2107827709018116096
n/a2107763728446394368
n/a2107701758112305152
n/a2107652362595209216
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
e2e116.67%1 of 6 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 1: 0 · Oct 2: 2 · Oct 3: 5 · Oct 4: 0 · Oct 5: 5 · Oct 6: 5 · Oct 7: 5PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x27f249e4c7e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-w7l7gcf22jkn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x27f249e4c7e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-w7l7gcf22jkn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1512977bb1a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qrbwmhc6tvhs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1512977bb1a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qrbwmhc6tvhs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x3dbe3c0729c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-g2tcvfj4mtzb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x3dbe3c0729c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-g2tcvfj4mtzb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107969374252961792
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e583.33%5 of 6 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 0 · Oct 7: 8DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x26a27898a0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-m9xrgvjx8lwn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "name": "e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "status": "Failed",
      "startTime": "2026-10-07T23:17:33.092346932Z",
      "endTime": "2026-10-07T23:46:38.337633525Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-whw2v-lppqg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-whw2v-lppqg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x26a27898a0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-m9xrgvjx8lwn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "name": "e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "status": "Failed",
      "startTime": "2026-10-07T23:17:33.092346932Z",
      "endTime": "2026-10-07T23:46:38.337633525Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-whw2v-lppqg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-whw2v-lppqg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x390344d20c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4wvdmxwppl4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80",
      "name": "02042050-d676-4d90-bee1-de9c86994c80",
      "status": "Failed",
      "startTime": "2026-10-07T14:30:29.815037524Z",
      "endTime": "2026-10-07T14:59:36.125232922Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x390344d20c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4wvdmxwppl4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80",
      "name": "02042050-d676-4d90-bee1-de9c86994c80",
      "status": "Failed",
      "startTime": "2026-10-07T14:30:29.815037524Z",
      "endTime": "2026-10-07T14:59:36.125232922Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xb98d7c3a108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6z9fvf852z99" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93cf84b2-52d0-403e-b5be-b66df90e2622
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93cf84b2-52d0-403e-b5be-b66df90e2622",
      "name": "93cf84b2-52d0-403e-b5be-b66df90e2622",
      "status": "Failed",
      "startTime": "2026-10-07T14:30:19.676644212Z",
      "endTime": "2026-10-07T14:59:26.138957659Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-m8jkv-j7ddl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6z9fvf852z99--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m8jkv-j7ddl to exist\nMachine agg-cluster-agg-np-a-m8jkv-lsbmq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-m8jkv-lsbmq: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-m8jkv-j7ddl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6z9fvf852z99--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m8jkv-j7ddl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xb98d7c3a108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6z9fvf852z99" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93cf84b2-52d0-403e-b5be-b66df90e2622
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93cf84b2-52d0-403e-b5be-b66df90e2622",
      "name": "93cf84b2-52d0-403e-b5be-b66df90e2622",
      "status": "Failed",
      "startTime": "2026-10-07T14:30:19.676644212Z",
      "endTime": "2026-10-07T14:59:26.138957659Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-m8jkv-j7ddl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6z9fvf852z99--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m8jkv-j7ddl to exist\nMachine agg-cluster-agg-np-a-m8jkv-lsbmq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-m8jkv-lsbmq: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-m8jkv-j7ddl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6z9fvf852z99--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m8jkv-j7ddl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2107969374252961792
n/a2107827709018116096
n/a2107763728446394368
n/a2107701758112305152
n/a2107652362595209216
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e583.33%5 of 6 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 5DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x115d2dac76c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-knjdfr9bnmx4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "name": "3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "status": "Failed",
      "startTime": "2026-10-07T23:45:51.426276166Z",
      "endTime": "2026-10-08T00:14:52.690397112Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x115d2dac76c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-knjdfr9bnmx4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "name": "3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "status": "Failed",
      "startTime": "2026-10-07T23:45:51.426276166Z",
      "endTime": "2026-10-08T00:14:52.690397112Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x25e8dd52bee0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-tkm2qrpq9mx2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4716e73-b072-461b-9a76-d43c0f1198b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4716e73-b072-461b-9a76-d43c0f1198b7",
      "name": "b4716e73-b072-461b-9a76-d43c0f1198b7",
      "status": "Failed",
      "startTime": "2026-10-07T15:25:44.42195423Z",
      "endTime": "2026-10-07T15:54:45.869722515Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97 to exist\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97 to exist\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x25e8dd52bee0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-tkm2qrpq9mx2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4716e73-b072-461b-9a76-d43c0f1198b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4716e73-b072-461b-9a76-d43c0f1198b7",
      "name": "b4716e73-b072-461b-9a76-d43c0f1198b7",
      "status": "Failed",
      "startTime": "2026-10-07T15:25:44.42195423Z",
      "endTime": "2026-10-07T15:54:45.869722515Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97 to exist\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-5nn97 to exist\nMachine q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-tkm2qrpq9mx2--managed/providers/Microsoft.Compute/virtualMachines/q5l6c5w9a6d0j2h-np-delete-main-vkt28-fd6j4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x28b27f4fd8c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-flmg4dn4tndz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8023a4bb-83f8-470d-8d09-d112e1caa401
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8023a4bb-83f8-470d-8d09-d112e1caa401",
      "name": "8023a4bb-83f8-470d-8d09-d112e1caa401",
      "status": "Failed",
      "startTime": "2026-10-07T11:07:13.045361342Z",
      "endTime": "2026-10-07T11:36:15.865514346Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz to exist\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz to exist\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x28b27f4fd8c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-flmg4dn4tndz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8023a4bb-83f8-470d-8d09-d112e1caa401
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8023a4bb-83f8-470d-8d09-d112e1caa401",
      "name": "8023a4bb-83f8-470d-8d09-d112e1caa401",
      "status": "Failed",
      "startTime": "2026-10-07T11:07:13.045361342Z",
      "endTime": "2026-10-07T11:36:15.865514346Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz to exist\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-9sqcz to exist\nMachine v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-flmg4dn4tndz--managed/providers/Microsoft.Compute/virtualMachines/v2z1k6a9o7e9k0r-np-deletable-qwwz5-bknr5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2107969374252961792
n/a2107827709018116096
n/a2107763728446394368
n/a2107701758112305152
n/a2107652362595209216
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down cluster resources, managed cluster, hosted cluster, namespaces; [descendantResources] remaining resources: applydesires, externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e116.67%1 of 6 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27901568a690>: 
    resource group "external-auth-cluster-ch86qj77dnn4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-ch86qj77dnn4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87",
      "name": "0cf8be7c-97b3-497d-b733-73f64068db87",
      "status": "Failed",
      "startTime": "2026-10-07T06:22:44.810261014Z",
      "endTime": "2026-10-07T06:46:52.177879454Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0x279015820520>{
                msg: "resource group \"external-auth-cluster-ch86qj77dnn4\": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0x279015820460>{
                    msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*fmt.wrapError | 0x279015820440>{
                        msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                        err: <*fmt.wrapError | 0x279015820420>{
                            msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            err: <*exported.ResponseError | 0x27901606a960>{
                                ErrorCode: "InternalServerError",
                                StatusCode: 200,
                                RawResponse: {
                                    Status: "200 OK",
                                    StatusCode: 200,
                                    Proto: "HTTP/2.0",
                                    ProtoMajor: 2,
                                    ProtoMinor: 0,
                                    Header: {
                                        "Cache-Control": [...],
                                        "X-Envoy-Upstream-Service-Time": [...],
                                        "X-Ms-Operation-Identifier": [...],
                                        "X-Ms-Correlation-Request-Id": [...],
                                        "X-Msedge-Ref": [...],
                                        "Pragma": [...],
                                        "Content-Type": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": [...],
                                        "Strict-Transport-Security": [...],
                                        "X-Content-Type-Options": [...],
                                        "Date": [...],
                                        "Content-Length": [...],
                                        "X-Ms-Request-Id": [...],
                                        "X-Ms-Routing-Request-Id": [...],
                                        "X-Cache": [...],
                                        "Expires": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": [...],
                                    },
                                    Body: <*exported.nopClosingBytesReader | 0x279015820220>{
                                        s: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        i: 0,
                                    },
                                    ContentLength: 1273,
                                    TransferEncoding: nil,
                                    Close: false,
                                    Uncompressed: false,
                                    Trailer: nil,
                                    Request: {
                                        Method: "GET",
                                        URL: {Scheme: ..., Opaque: ..., User: ..., Host: ..., Path: ..., Fragment: ..., RawQuery: ..., RawPath: ..., RawFragment: ..., ForceQuery: ..., OmitHost: ...},
                                        Proto: "HTTP/1.1",
                                        ProtoMajor: 1,
                                        ProtoMinor: 1,
                                        Header: {...: ..., ...: ...},
                                        Body: nil,
                                        GetBody: nil,
                                        ContentLength: 0,
                                        TransferEncoding: nil,
                                        Close: false,
                                        Host: "management.azure.com",
                                        Form: nil,
                                        PostForm: nil,
                                        MultipartForm: nil,
                                        Trailer: nil,
                                        RemoteAddr: "",
                                        RequestURI: "",
                                        TLS: nil,
                                        Cancel: nil,
                                        Response: nil,
                                        Pattern: "",
                                        ctx: <suppressed context>,
                                        pat: nil,
                                        matches: nil,
                                        otherValues: nil,
                                    },
                                    TLS: {
                                        Version: 772,
                                        HandshakeComplete: true,
                                        DidResume: false,
                                        CipherSuite: 4866,
                                        CurveID: 25,
                                        NegotiatedProtocol: "h2",
                                        NegotiatedProtocolIsMutual: true,
                                        ServerName: "management.azure.com",
                                        PeerCertificates: [..., ..., ...],
                                        VerifiedChains: [...],
                                        SignedCertificateTimestamps: nil,
                                        OCSPResponse: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        TLSUnique: nil,
                                        ECHAccepted: false,
                                        HelloRetryRequest: true,
                                        ekm: 0x585e900,
                                        testingOnlyPeerSignatureAlgorithm: 2052,
                                    },
                                },
                                errMsg: "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27901568a690>: 
    resource group "external-auth-cluster-ch86qj77dnn4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-ch86qj77dnn4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87",
      "name": "0cf8be7c-97b3-497d-b733-73f64068db87",
      "status": "Failed",
      "startTime": "2026-10-07T06:22:44.810261014Z",
      "endTime": "2026-10-07T06:46:52.177879454Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0x279015820520>{
                msg: "resource group \"external-auth-cluster-ch86qj77dnn4\": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0x279015820460>{
                    msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*fmt.wrapError | 0x279015820440>{
                        msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                        err: <*fmt.wrapError | 0x279015820420>{
                            msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            err: <*exported.ResponseError | 0x27901606a960>{
                                ErrorCode: "InternalServerError",
                                StatusCode: 200,
                                RawResponse: {
                                    Status: "200 OK",
                                    StatusCode: 200,
                                    Proto: "HTTP/2.0",
                                    ProtoMajor: 2,
                                    ProtoMinor: 0,
                                    Header: {
                                        "Cache-Control": [...],
                                        "X-Envoy-Upstream-Service-Time": [...],
                                        "X-Ms-Operation-Identifier": [...],
                                        "X-Ms-Correlation-Request-Id": [...],
                                        "X-Msedge-Ref": [...],
                                        "Pragma": [...],
                                        "Content-Type": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": [...],
                                        "Strict-Transport-Security": [...],
                                        "X-Content-Type-Options": [...],
                                        "Date": [...],
                                        "Content-Length": [...],
                                        "X-Ms-Request-Id": [...],
                                        "X-Ms-Routing-Request-Id": [...],
                                        "X-Cache": [...],
                                        "Expires": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": [...],
                                    },
                                    Body: <*exported.nopClosingBytesReader | 0x279015820220>{
                                        s: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        i: 0,
                                    },
                                    ContentLength: 1273,
                                    TransferEncoding: nil,
                                    Close: false,
                                    Uncompressed: false,
                                    Trailer: nil,
                                    Request: {
                                        Method: "GET",
                                        URL: {Scheme: ..., Opaque: ..., User: ..., Host: ..., Path: ..., Fragment: ..., RawQuery: ..., RawPath: ..., RawFragment: ..., ForceQuery: ..., OmitHost: ...},
                                        Proto: "HTTP/1.1",
                                        ProtoMajor: 1,
                                        ProtoMinor: 1,
                                        Header: {...: ..., ...: ...},
                                        Body: nil,
                                        GetBody: nil,
                                        ContentLength: 0,
                                        TransferEncoding: nil,
                                        Close: false,
                                        Host: "management.azure.com",
                                        Form: nil,
                                        PostForm: nil,
                                        MultipartForm: nil,
                                        Trailer: nil,
                                        RemoteAddr: "",
                                        RequestURI: "",
                                        TLS: nil,
                                        Cancel: nil,
                                        Response: nil,
                                        Pattern: "",
                                        ctx: <suppressed context>,
                                        pat: nil,
                                        matches: nil,
                                        otherValues: nil,
                                    },
                                    TLS: {
                                        Version: 772,
                                        HandshakeComplete: true,
                                        DidResume: false,
                                        CipherSuite: 4866,
                                        CurveID: 25,
                                        NegotiatedProtocol: "h2",
                                        NegotiatedProtocolIsMutual: true,
                                        ServerName: "management.azure.com",
                                        PeerCertificates: [..., ..., ...],
                                        VerifiedChains: [...],
                                        SignedCertificateTimestamps: nil,
                                        OCSPResponse: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        TLSUnique: nil,
                                        ECHAccepted: false,
                                        HelloRetryRequest: true,
                                        ekm: 0x585e900,
                                        testingOnlyPeerSignatureAlgorithm: 2052,
                                    },
                                },
                                errMsg: "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107701758112305152
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>
full failure pattern:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e116.67%1 of 6 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 1: 0 · Oct 2: 0 · Oct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x2655580ad6c0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x2655580ad6c0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107827709018116096