CIHealth Failure Patterns

Environment: DEV

Failure patterns: 51 · Failures matched: 425 · Total runs: 71 · Runs affected: 55

Failure pattern

Failed at

Trend

Also in

ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsof...

full failure pattern:

ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x295f3df44be0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-f56729plnfx5": PUT https://rp.j6769152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-f56729plnfx5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x295f3df44be0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-f56729plnfx5": PUT https://rp.j6769152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-f56729plnfx5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x156cf26d7900>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-j4cvjbg6jcxh": PUT https://rp.j0287232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j4cvjbg6jcxh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x156cf26d7900>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-j4cvjbg6jcxh": PUT https://rp.j0287232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j4cvjbg6jcxh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 03:55	PR #7397	2107681070840287232

failed to create nodepool <nodepool>: failed starting nodepool creation "np-1" for cluster "<cluster>" in resourcegroup="<resource-group>": PUT <url>

full failure pattern:

failed to create nodepool <nodepool>: failed starting nodepool creation "np-1" for cluster "<cluster>" in resourcegroup="<resource-group>": PUT <url>

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e182923c240>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ls445v45mtp4": PUT https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-ls445v45mtp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e182923c240>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ls445v45mtp4": PUT https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-ls445v45mtp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:162]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e9eea24a6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fh65w65xjrlv": PUT https://rp.j8091008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fh65w65xjrlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:162]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e9eea24a6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fh65w65xjrlv": PUT https://rp.j8091008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fh65w65xjrlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.1	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 12:57	PR #7325	2107817608098091008

ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])

full failure pattern:

ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:142]: Timed out after 1200.007s.
Insights Operator did not reach Disabled/NoToken state, the NoToken-to-healthy transition cannot be tested
Expected success, but got an error:
    <*errors.errorString | 0x27811265a7d0>: 
    ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
    {
        s: "ClusterOperator \"insights\" not yet in Disabled/NoToken state (conditions: [])",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:142]: Timed out after 1200.007s.
Insights Operator did not reach Disabled/NoToken state, the NoToken-to-healthy transition cannot be tested
Expected success, but got an error:
    <*errors.errorString | 0x27811265a7d0>: 
    ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
    {
        s: "ClusterOperator \"insights\" not yet in Disabled/NoToken state (conditions: [])",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to see an HCP cluster registered with console.redhat.com via the Insights Operator	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 21:02	PR #6954	2107939651057618944

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3974085580d8>: 
    resource group "external-auth-cluster-jkpp8mdvzxsh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-jkpp8mdvzxsh" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac",
      "name": "7ef42875-c74c-425c-8fe8-0909778dceac",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:50.350274496Z",
      "endTime": "2026-10-07T18:24:57.82988818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3adt66dtiffarpvt6smilc2f19037 still exists (deletion dispatched at 2026-10-07T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3974085580d8>: 
    resource group "external-auth-cluster-jkpp8mdvzxsh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-jkpp8mdvzxsh" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac",
      "name": "7ef42875-c74c-425c-8fe8-0909778dceac",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:50.350274496Z",
      "endTime": "2026-10-07T18:24:57.82988818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3adt66dtiffarpvt6smilc2f19037 still exists (deletion dispatched at 2026-10-07T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:49	PR #7143	2107875990943305728

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2183e1e34210>: 
    resource group "private-ingress-76lkp46mpc8d": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-76lkp46mpc8d" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "name": "1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "status": "Failed",
      "startTime": "2026-10-07T18:03:40.469898232Z",
      "endTime": "2026-10-07T18:27:41.300163086Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cd0prbd7pkt4n50qh2rsnhp0bjo4 still exists (deletion dispatched at 2026-10-07T18:03:41Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2183e1e34210>: 
    resource group "private-ingress-76lkp46mpc8d": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-76lkp46mpc8d" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "name": "1f6732c1-e698-4577-a105-6f4bedb6c48b",
      "status": "Failed",
      "startTime": "2026-10-07T18:03:40.469898232Z",
      "endTime": "2026-10-07T18:27:41.300163086Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cd0prbd7pkt4n50qh2rsnhp0bjo4 still exists (deletion dispatched at 2026-10-07T18:03:41Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:47	PR #7400	2107875420446658560

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f397df201c8>: 
    resource group "clusternp128-89h59bcbsmnq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-89h59bcbsmnq" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "name": "84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:09.863415259Z",
      "endTime": "2026-10-07T18:24:11.307794702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3chch61ipf94injnrljj1hk0caf92 still exists (deletion dispatched at 2026-10-07T18:00:10Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f397df201c8>: 
    resource group "clusternp128-89h59bcbsmnq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-89h59bcbsmnq" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "name": "84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:09.863415259Z",
      "endTime": "2026-10-07T18:24:11.307794702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3chch61ipf94injnrljj1hk0caf92 still exists (deletion dispatched at 2026-10-07T18:00:10Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:47	PR #7400	2107875420446658560

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderCluster...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xeda890f47f8>: 
    resource group "oidc-wi-h74rjsb4b6b4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-h74rjsb4b6b4" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98",
      "name": "347d2fde-a5b9-4245-8439-df8aaab88f98",
      "status": "Failed",
      "startTime": "2026-10-07T18:11:44.745635305Z",
      "endTime": "2026-10-07T18:35:51.763179185Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xeda890f47f8>: 
    resource group "oidc-wi-h74rjsb4b6b4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-h74rjsb4b6b4" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/347d2fde-a5b9-4245-8439-df8aaab88f98",
      "name": "347d2fde-a5b9-4245-8439-df8aaab88f98",
      "status": "Failed",
      "startTime": "2026-10-07T18:11:44.745635305Z",
      "endTime": "2026-10-07T18:35:51.763179185Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:49	PR #7143	2107875990943305728

Internal error occurred: failed calling webhook "webhook

full failure pattern:

Internal error occurred: failed calling webhook "webhook

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:337]: Timed out after 120.000s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
    <*errors.StatusError | 0x3dfc585d0640>: 
    Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:337]: Timed out after 120.000s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
    <*errors.StatusError | 0x3dfc585d0640>: 
    Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 08:29	PR #7370	2107750168211230720

VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout

full failure pattern:

VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:128]: failed to verify HCP cluster viability
Unexpected error:
    <*errors.joinError | 0x3fa9a7564d98>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0x3fa9a6cff3e0>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6cff3c0>{
                    msg: "failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a7398db0>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0x3fa9a716a690>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3fa9a6ee5c60>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6ee5c40>{
                    msg: "failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a786a420>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0x3fa9a7c7d0e0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:128]: failed to verify HCP cluster viability
Unexpected error:
    <*errors.joinError | 0x3fa9a7564d98>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0x3fa9a6cff3e0>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6cff3c0>{
                    msg: "failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a7398db0>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0x3fa9a716a690>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3fa9a6ee5c60>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6ee5c40>{
                    msg: "failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a786a420>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0x3fa9a7c7d0e0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 03:35	PR #7400	2107676117312737280

VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-<id> on node <node> after retries: a...

full failure pattern:

VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-<id> on node <node> after retries: an error on the server ("unknown") has prevented the request from succeeding

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xf6f46a980d8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-tkqcj)
    {
        errs: [
            <*fmt.wrapError | 0xf6f45174120>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                err: <*fmt.wrapError | 0xf6f45174100>{
                    msg: "failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                    err: <*fmt.wrapError | 0xf6f451742a0>{
                        msg: "failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                        err: <*errors.StatusError | 0xf6f4458f7c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-tkqcj",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xf6f46a980d8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-tkqcj)
    {
        errs: [
            <*fmt.wrapError | 0xf6f45174120>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                err: <*fmt.wrapError | 0xf6f45174100>{
                    msg: "failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                    err: <*fmt.wrapError | 0xf6f451742a0>{
                        msg: "failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                        err: <*errors.StatusError | 0xf6f4458f7c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-tkqcj",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 19:32	PR #7325	2107916955439599616

VerifyHyperShiftIngressCertificate timed out after 10m0

full failure pattern:

VerifyHyperShiftIngressCertificate timed out after 10m0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 23:05	PR #7325	2107970662109810688

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0x339d2c183220>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);
    let _endTime = datetime(2026-10-07T22:32:32.2954057Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-n8d6fgpt4qrq'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T22:32:43.6043688Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd", ActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", SubActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityType:"GW.Http.CallContext", ParentActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityStack:"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x339d2b3c97a0>{
            errs: [
                <*fmt.wrapError | 0x339d2c183200>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x339d2c1831e0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x339d2c1830c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x339d2c183080>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x339d2ae3d200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0x339d2c183220>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);
    let _endTime = datetime(2026-10-07T22:32:32.2954057Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-n8d6fgpt4qrq'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T22:32:43.6043688Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd", ActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", SubActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityType:"GW.Http.CallContext", ParentActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityStack:"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x339d2b3c97a0>{
            errs: [
                <*fmt.wrapError | 0x339d2c183200>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x339d2c1831e0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x339d2c1830c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x339d2c183080>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x339d2ae3d200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 21:02	PR #6954	2107939651057618944

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.006s.
Expected success, but got an error:
    <*fmt.wrapError | 0x11eea31d1d40>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);
    let _endTime = datetime(2026-10-08T00:23:30.3337686Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7nt846zks6zp'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T00:23:37.3079994Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d", ActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", SubActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityType:"GW.Http.CallContext", ParentActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityStack:"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x11eea2518d38>{
            errs: [
                <*fmt.wrapError | 0x11eea31d1d20>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x11eea31d1d00>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x11eea2cd05c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x11eea2cd0560>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x11eea1622000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.006s.
Expected success, but got an error:
    <*fmt.wrapError | 0x11eea31d1d40>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);
    let _endTime = datetime(2026-10-08T00:23:30.3337686Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7nt846zks6zp'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T00:23:37.3079994Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d", ActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", SubActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityType:"GW.Http.CallContext", ParentActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityStack:"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x11eea2518d38>{
            errs: [
                <*fmt.wrapError | 0x11eea31d1d20>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x11eea31d1d00>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x11eea2cd05c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x11eea2cd0560>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x11eea1622000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 23:00	PR #6928	2107969278706716672

failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline...

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-07T14:40:32.913Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-10-07T14:40:33.412Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157 global:map[imageOverrides:map[governance_policy_propagator:governance-policy-propagator-acm-216@sha256:deb737b19932d520553c902ee5e0738ebeaad4592e8aa9649c41fec6fc00b62e] registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-10-07T14:40:33.412Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:40:33.440Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 namespace=multicluster-engine
time=2026-10-07T14:40:33.491Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 dryRun=false
time=2026-10-07T14:40:33.491Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:40:33.534Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:51:01.640Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 err="failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-10-07T14:51:01.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 13:55	PR #7397	2107832305794945024

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T00:05:01Z
  Ended:       2026-10-08T00:11:00Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6670976-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T23:44:13Z
  Ended:       2026-10-07T23:58:13Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6213760-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T21:16:57Z
  Ended:       2026-10-07T21:22:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3227776-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	29

Affected runs (29)

Date (UTC)	Associated PR	Prow job
2026-10-07 22:55	PR #7417	2107967978476670976
2026-10-07 22:42	PR #7414	2107964840596213760
2026-10-07 20:10	PR #6364	2107926490833227776
2026-10-07 20:09	PR #7305	2107926353130033152
2026-10-07 19:30	PR #7414	2107916465603612672
2026-10-07 19:20	PR #7412	2107914072107257856
2026-10-07 18:41	PR #7288	2107904235558408192
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:48	PR #7322	2107875761300967424
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:37	PR #7310	2107872947002675200
2026-10-07 14:42	PR #7410	2107844183950626816
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 11:55	PR #7405	2107802095104036864
2026-10-07 10:22	PR #6688	2107778619966230528
2026-10-07 08:38	PR #7133	2107752436901875712
2026-10-07 08:36	PR #7403	2107751899905134592
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7143	2107750854802018304
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 08:29	PR #7370	2107750168211230720
2026-10-07 08:29	PR #6968	2107750153682161664
2026-10-07 07:47	PR #7310	2107739528360890368
2026-10-07 03:55	PR #7397	2107681070840287232
2026-10-07 03:51	PR #7097	2107680144293367808

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x2884a5492ee0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dk2zvdznnrzn" to finish creating: GET https://rp.j7618944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "name": "9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "status": "Failed",
      "startTime": "2026-10-07T22:28:55.765188822Z",
      "endTime": "2026-10-07T22:57:56.06793791Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x2884a5492ee0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dk2zvdznnrzn" to finish creating: GET https://rp.j7618944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "name": "9a5a009b-1b3a-4c53-8483-6562b10f6eb0",
      "status": "Failed",
      "startTime": "2026-10-07T22:28:55.765188822Z",
      "endTime": "2026-10-07T22:57:56.06793791Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dk2zvdznnrzn--managed/providers/Microsoft.Compute/virtualMachines/r1v2t1h5o2n7k4l-np-1-kbxtb-qjzfl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm"
Unexpected error:
    <*fmt.wrapError | 0x2aa4ed1a5b40>: 
    failed to create NodePool np-ckh4xm: failed waiting for nodepool="np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm" in resourcegroup="rg-candidate-4-23-g2p8crd8t6kf" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7",
      "name": "7981efbe-9393-4366-bd69-e17288ac49a7",
      "status": "Failed",
      "startTime": "2026-10-07T20:35:08.39379338Z",
      "endTime": "2026-10-07T21:04:16.844016594Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm"
Unexpected error:
    <*fmt.wrapError | 0x2aa4ed1a5b40>: 
    failed to create NodePool np-ckh4xm: failed waiting for nodepool="np-ckh4xm" for cluster "cluster-candidate-4-23-ckh4xm" in resourcegroup="rg-candidate-4-23-g2p8crd8t6kf" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7981efbe-9393-4366-bd69-e17288ac49a7",
      "name": "7981efbe-9393-4366-bd69-e17288ac49a7",
      "status": "Failed",
      "startTime": "2026-10-07T20:35:08.39379338Z",
      "endTime": "2026-10-07T21:04:16.844016594Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-7v82k to exist\nMachine l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-g2p8crd8t6kf-candidate-ckh4xm--managed/providers/Microsoft.Compute/virtualMachines/l0u9z2p5c7y8d4r-np-ckh4xm-5hj7l-lx6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq"
Unexpected error:
    <*fmt.wrapError | 0x99848406080>: 
    failed to create NodePool np-mcx5bq: failed waiting for nodepool="np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq" in resourcegroup="rg-candidate-5-1-tvjcv9r9dthd" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d",
      "name": "fe8bd558-9f50-442b-bfc7-12311927703d",
      "status": "Failed",
      "startTime": "2026-10-07T18:23:04.012039216Z",
      "endTime": "2026-10-07T18:52:11.882573758Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq"
Unexpected error:
    <*fmt.wrapError | 0x99848406080>: 
    failed to create NodePool np-mcx5bq: failed waiting for nodepool="np-mcx5bq" for cluster "cluster-candidate-5-1-mcx5bq" in resourcegroup="rg-candidate-5-1-tvjcv9r9dthd" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe8bd558-9f50-442b-bfc7-12311927703d",
      "name": "fe8bd558-9f50-442b-bfc7-12311927703d",
      "status": "Failed",
      "startTime": "2026-10-07T18:23:04.012039216Z",
      "endTime": "2026-10-07T18:52:11.882573758Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-ft2sr to exist\nMachine q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvjcv9r9dthd-candidate-mcx5bq--managed/providers/Microsoft.Compute/virtualMachines/q2k6t7c1l5r1b0v-np-mcx5bq-wpgvp-pfw7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (20)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	13
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	13
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	12
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	11
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	11
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	11
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	11
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	10
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	10
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	9
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with a private key vault using an available API version	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a fully private cluster with private KAS and private ingress and verify both are only reachable from VNet	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a fully private cluster with private KAS, private ingress, and private KeyVault and verify all are operational	1

Affected runs (17)

Date (UTC)	Associated PR	Prow job
2026-10-07 21:02	PR #6954	2107939651057618944
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 10:22	PR #6688	2107778619966230528
2026-10-07 09:31	PR #7405	2107765914286952448
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 07:47	PR #7310	2107739528360890368
2026-10-07 03:55	PR #7397	2107681070840287232

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x259bbf69a258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6m6qldl4t7ln" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "name": "74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "status": "Failed",
      "startTime": "2026-10-07T18:16:07.02818237Z",
      "endTime": "2026-10-07T18:45:11.860346136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x259bbf69a258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6m6qldl4t7ln" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "name": "74400ec9-b7cc-4ceb-933e-5f049cfbef5c",
      "status": "Failed",
      "startTime": "2026-10-07T18:16:07.02818237Z",
      "endTime": "2026-10-07T18:45:11.860346136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-72bgn-5qwj8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-5qwj8 to exist\nMachine agg-cluster-agg-np-a-72bgn-msjpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6m6qldl4t7ln--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-72bgn-msjpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x14bbdf4ea0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-db8t5zc8d27b" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b",
      "name": "28c778fc-2439-43ac-8817-96efd7f98d9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:23.837365353Z",
      "endTime": "2026-10-07T18:49:29.442401902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x14bbdf4ea0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-db8t5zc8d27b" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/28c778fc-2439-43ac-8817-96efd7f98d9b",
      "name": "28c778fc-2439-43ac-8817-96efd7f98d9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:23.837365353Z",
      "endTime": "2026-10-07T18:49:29.442401902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-sxrv8-whnvh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-sxrv8-p75dc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-db8t5zc8d27b--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-sxrv8-p75dc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	9
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	7

Affected runs (12)

Date (UTC)	Associated PR	Prow job
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 09:31	PR #7405	2107765914286952448
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 03:55	PR #7397	2107681070840287232

ERROR CODE: DeadlineExceeded; provider Microsoft.Network

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (11)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	5
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (10)

Date (UTC)	Associated PR	Prow job
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T21:50:06Z
  Ended:       2026-10-07T21:56:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3227776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta6nelbkbeh1s64mru6t7ohgsjall8a-h3w3a5j6l2a4p9s", pod="openshift-apiserver-d5448bdcd-ltz5s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ta6nelbkbeh1s64mru6t7ohgsjall8a-h3w3a5j6l2a4p9s/openshift-apiserver-d5448bdcd-ltz5s has been in a non-ready state for longer than 5 minutes.

alert fired 9 time(s) (2 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T20:35:02Z
  Ended:       2026-10-07T20:41:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7631744-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta5emb000bsfom310889hpe5casu1jr-v5k9k8q2w5x0q6v", pod="openshift-apiserver-b9fcdfdb5-55mlh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ta5emb000bsfom310889hpe5casu1jr-v5k9k8q2w5x0q6v/openshift-apiserver-b9fcdfdb5-55mlh has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-07T20:35:02Z
  Ended:       2026-10-07T20:41:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7631744-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta5fdlm2qo2eb32ju5uasrq631306b1-cilium-cluster", pod="multus-admission-controller-578b7846f7-h57x6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ta5fdlm2qo2eb32ju5uasrq631306b1-cilium-cluster/multus-admission-controller-578b7846f7-h57x6 has been in a non-ready state for longer than 5 minutes.

alert fired 6 time(s) (3 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T18:11:35Z
  Ended:       2026-10-07T18:21:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3880064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta3l68gqrcq7gs6igibvm68inmua3g6-u0t5l5t9k7x3r9l", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ta3l68gqrcq7gs6igibvm68inmua3g6-u0t5l5t9k7x3r9l/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-07T18:11:35Z
  Ended:       2026-10-07T18:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3880064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta3kutmkbpo2kifvv50cfrikfbop2vu-y5c5w0x7t8g5d0d", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ta3kutmkbpo2kifvv50cfrikfbop2vu-y5c5w0x7t8g5d0d/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-07T18:12:33Z
  Ended:       2026-10-07T18:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3880064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ta3l4ibb8m6kaqmai8gv3qag7e9qrug-t0m8d4r3h1i2g1k", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ta3l4ibb8m6kaqmai8gv3qag7e9qrug-t0m8d4r3h1i2g1k/etcd-1 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	9

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-10-07 20:10	PR #6364	2107926490833227776
2026-10-07 18:59	PR #7399	2107908786537631744
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:04	PR #7411	2107879778164936704
2026-10-07 14:42	PR #7410	2107844183950626816
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 06:18	PR #7402	2107717187903950848
2026-10-07 03:35	PR #7400	2107676117312737280

alert [svc] KustoLogsDataStale fired

full failure pattern:

alert [svc] KustoLogsDataStale fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T02:13:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5347456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.24:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347456/providers/microsoft.monitor/accounts/services-j5347456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8fb68546b-b5vcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5347456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T02:14:49Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5347456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.24:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347456/providers/microsoft.monitor/accounts/services-j5347456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8fb68546b-b5vcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5347456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T19:18:35Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j6440704-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.112:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6440704/providers/microsoft.monitor/accounts/services-j6440704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6f8d84d4d4-wkcql", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j6440704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T19:18:35Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j6440704-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.112:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6440704/providers/microsoft.monitor/accounts/services-j6440704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6f8d84d4d4-wkcql", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j6440704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:35:53Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9974528-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9974528/providers/microsoft.monitor/accounts/services-j9974528", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6c57d99f8b-ljtj9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9974528-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:35:53Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9974528-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9974528/providers/microsoft.monitor/accounts/services-j9974528", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6c57d99f8b-ljtj9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9974528-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-07 23:41	PR #7288	2107979694195347456
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:34:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8893056-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.61:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8893056/providers/microsoft.monitor/accounts/hcps-j8893056", namespace="ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-n8krx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:34:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8893056-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.61:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8893056/providers/microsoft.monitor/accounts/hcps-j8893056", namespace="ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-n8krx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:29:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3305728-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3305728/providers/microsoft.monitor/accounts/hcps-j3305728", namespace="ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gmh4h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:29:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3305728-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3305728/providers/microsoft.monitor/accounts/hcps-j3305728", namespace="ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gmh4h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ta38bnq43hvrce83gic7njl84d30rep-i1r8s9j1s0i1f3t/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:49	PR #7143	2107875990943305728

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:02:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-c4d6b786-bgkk8", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4d6b786-bgkk8 has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T02:42:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5628416-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-445sp", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-445sp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T02:43:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5628416-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-gpz7w", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-gpz7w has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 22:28	PR #7305	2107961335642853376
2026-10-07 01:40	PR #7397	2107647281405628416

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T18:21:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8893056-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.61:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8893056/providers/microsoft.monitor/accounts/hcps-j8893056", namespace="ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics", pod="cloud-network-config-controller-8445d77f54-dbcgg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b63493d6-07ef-4acd-8041-60edf966639a"
  Description: Pod ocm-arohcpci01-2ta3f94m6csa7tur24re6meedru3s5o4-ksm-hcp-metrics/cloud-network-config-controller-8445d77f54-dbcgg (controller) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 17:04	PR #7414	2107879778638893056

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

full failure pattern:

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x1e187b528690>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x1e187b528690>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 03:51	PR #7097	2107680144293367808

alert [svc] MaestroAgentMetricsProxyDown fired

full failure pattern:

alert [svc] MaestroAgentMetricsProxyDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T02:35:12Z
  Ended:       2026-10-07T02:42:11Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5628416-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 01:40	PR #7397	2107647281405628416

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x37d635eafd40>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lpkn85dzg6wg" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191",
      "name": "321127fe-bb12-4e32-b036-dd251357b191",
      "status": "Failed",
      "startTime": "2026-10-07T20:56:46.02075406Z",
      "endTime": "2026-10-07T21:25:46.806941683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x37d635eafd40>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lpkn85dzg6wg" to finish creating: GET https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/321127fe-bb12-4e32-b036-dd251357b191",
      "name": "321127fe-bb12-4e32-b036-dd251357b191",
      "status": "Failed",
      "startTime": "2026-10-07T20:56:46.02075406Z",
      "endTime": "2026-10-07T21:25:46.806941683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lpkn85dzg6wg--managed/providers/Microsoft.Compute/virtualMachines/m3p5o0q6k9u0x5e-np-delete-main-kckl6-dqrv2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2a2d2cf153c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qtdjcjq4g46d" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "name": "d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "status": "Failed",
      "startTime": "2026-10-07T18:35:04.887969322Z",
      "endTime": "2026-10-07T19:04:11.73701564Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2a2d2cf153c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qtdjcjq4g46d" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "name": "d8ae7058-dd29-49ab-b721-9f8daeb7b0c8",
      "status": "Failed",
      "startTime": "2026-10-07T18:35:04.887969322Z",
      "endTime": "2026-10-07T19:04:11.73701564Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-6xk2g to exist\nMachine c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qtdjcjq4g46d--managed/providers/Microsoft.Compute/virtualMachines/c1m0g4d8l1a5k7p-np-deletable-d2gzm-rjncx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xf6ccd9b14e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-b4g4j9tj9qn5" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "name": "6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "status": "Failed",
      "startTime": "2026-10-07T18:34:53.377476522Z",
      "endTime": "2026-10-07T19:03:57.309891257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xf6ccd9b14e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-b4g4j9tj9qn5" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "name": "6f1cecb5-6672-47d6-91bf-bf444c657b24",
      "status": "Failed",
      "startTime": "2026-10-07T18:34:53.377476522Z",
      "endTime": "2026-10-07T19:03:57.309891257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-l4bt8 to exist\nMachine w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-b4g4j9tj9qn5--managed/providers/Microsoft.Compute/virtualMachines/w8m9b4n1w5y7h6s-np-deletable-686hg-zqww5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	14
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (14)

Date (UTC)	Associated PR	Prow job
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 09:31	PR #7405	2107765914286952448
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 03:55	PR #7397	2107681070840287232

cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"

full failure pattern:

cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-zmb9zlf6bqmq/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-zmb9zlf6bqmq/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-w8w2hprnrv4g/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-w8w2hprnrv4g/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-cbtpkz6skzqd/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-cbtpkz6skzqd/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-10-07 20:18	PR #7415	2107928469479362560
2026-10-07 18:41	PR #7288	2107904235558408192
2026-10-07 14:14	PR #7227	2107836958540566528
2026-10-07 09:45	PR #7398	2107769248779603968
2026-10-07 07:47	PR #7310	2107739528360890368

VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil

full failure pattern:

VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e8a40f9d110>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e8a41056380>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e8a40efe990>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e8a40f9d110>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e8a41056380>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e8a40efe990>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x21f6a70e8e10>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x21f6a69d2ce0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x21f6a85f6620>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x21f6a70e8e10>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x21f6a69d2ce0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x21f6a85f6620>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x2b3cb6331890>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x2b3cb6202180>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x2b3cb57e7010>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x2b3cb6331890>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x2b3cb6202180>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x2b3cb57e7010>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-07 23:05	PR #7325	2107970662109810688
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

alert [svc] LeaderElectionLeaseStale fired

full failure pattern:

alert [svc] LeaderElectionLeaseStale fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T01:40:45Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1217408-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1217408/providers/microsoft.monitor/accounts/services-j1217408", namespace="klusterlet-2ta9fcfdh71dkv4knt12m1ug0ranvkol", pod="arohcp-monitor-kube-state-metrics-7988dc648c-58rr2", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta9fcfdh71dkv4knt12m1ug0ranvkol on cluster ci01-j1217408-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:39:35Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j6520192-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.219:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6520192/providers/microsoft.monitor/accounts/services-j6520192", namespace="klusterlet-2ta7n5c8id8chjml49ukhi22658fcgoj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-lkfg4", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta7n5c8id8chjml49ukhi22658fcgoj on cluster ci01-j6520192-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T21:31:10Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3612672-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3612672/providers/microsoft.monitor/accounts/services-j3612672", namespace="klusterlet-2ta5r5t5b117do0a5grfaj0nbn80qof7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zcpkv", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta5r5t5b117do0a5grfaj0nbn80qof7 on cluster ci01-j3612672-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T21:42:11Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3612672-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3612672/providers/microsoft.monitor/accounts/services-j3612672", namespace="klusterlet-2ta5r55d86kstjo2bp5n4d3bvb279pda", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zcpkv", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ta5r55d86kstjo2bp5n4d3bvb279pda on cluster ci01-j3612672-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-07 23:41	PR #7361	2107979674851217408
2026-10-07 22:04	PR #7412	2107955248256520192
2026-10-07 19:30	PR #7414	2107916465603612672

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-ff6jdg"
Unexpected error:
    <*fmt.wrapError | 0x17d44c09e520>: 
    DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x17d44c09e500>{
            msg: "DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-ff6jdg"
Unexpected error:
    <*fmt.wrapError | 0x17d44c09e520>: 
    DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x17d44c09e500>{
            msg: "DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-vdz5wq"
Unexpected error:
    <*fmt.wrapError | 0x68ffbb0c4e0>: 
    DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x68ffbb0c4c0>{
            msg: "DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-vdz5wq"
Unexpected error:
    <*fmt.wrapError | 0x68ffbb0c4e0>: 
    DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x68ffbb0c4c0>{
            msg: "DNS for agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5fr7p.apps.aro.v2a6f1z0z3t6f6o.v5i6.j0890368.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x2a03aec70780>: 
    DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x2a03aec70740>{
            msg: "DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x2a03aec70780>: 
    DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x2a03aec70740>{
            msg: "DNS for agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g9blz.apps.aro.u2q5q2w5k8k4z5y.u6hb.j3950848.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-07 22:55	PR #7417	2107967978476670976
2026-10-07 07:47	PR #7310	2107739528360890368
2026-10-07 06:18	PR #7402	2107717187903950848

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f8378088558>: 
    resource group "kusto-logs-zdcz6ngd7l8s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zdcz6ngd7l8s" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "name": "ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "status": "Failed",
      "startTime": "2026-10-07T13:55:49.637158017Z",
      "endTime": "2026-10-07T14:19:49.807578144Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vu3tto21eb5vs2oeuv4d035r4kh9q still exists (deletion dispatched at 2026-10-07T13:56:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f8378088558>: 
    resource group "kusto-logs-zdcz6ngd7l8s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zdcz6ngd7l8s" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "name": "ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "status": "Failed",
      "startTime": "2026-10-07T13:55:49.637158017Z",
      "endTime": "2026-10-07T14:19:49.807578144Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vu3tto21eb5vs2oeuv4d035r4kh9q still exists (deletion dispatched at 2026-10-07T13:56:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xf860122e060>: 
    resource group "rg-zstream-upgrade-4-21-d7rnngth6rmt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-s5wrnm" in resourcegroup="rg-zstream-upgrade-4-21-d7rnngth6rmt" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "name": "de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "status": "Failed",
      "startTime": "2026-10-07T14:14:15.375309734Z",
      "endTime": "2026-10-07T14:38:20.14126941Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtq0dj9uuqrc354smigdjo5gckvsa still exists (deletion dispatched at 2026-10-07T14:14:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xf860122e060>: 
    resource group "rg-zstream-upgrade-4-21-d7rnngth6rmt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-s5wrnm" in resourcegroup="rg-zstream-upgrade-4-21-d7rnngth6rmt" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "name": "de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "status": "Failed",
      "startTime": "2026-10-07T14:14:15.375309734Z",
      "endTime": "2026-10-07T14:38:20.14126941Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtq0dj9uuqrc354smigdjo5gckvsa still exists (deletion dispatched at 2026-10-07T14:14:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3d715b6dc0c0>: 
    resource group "admin-api-serialconsole-kgtb4656t7wx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-kgtb4656t7wx" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "name": "63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "status": "Failed",
      "startTime": "2026-10-07T14:05:26.627856798Z",
      "endTime": "2026-10-07T14:29:30.238184303Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtu5p75nk86f54m2uds6vreosd2p2 still exists (deletion dispatched at 2026-10-07T14:05:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3d715b6dc0c0>: 
    resource group "admin-api-serialconsole-kgtb4656t7wx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-kgtb4656t7wx" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "name": "63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "status": "Failed",
      "startTime": "2026-10-07T14:05:26.627856798Z",
      "endTime": "2026-10-07T14:29:30.238184303Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtu5p75nk86f54m2uds6vreosd2p2 still exists (deletion dispatched at 2026-10-07T14:05:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (68)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.21 minor to 4.22 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xf53ccb2c090>: 
    resource group "rg-cordon-test-zfbgp6td9bvg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-zfbgp6td9bvg" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "name": "3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "status": "Failed",
      "startTime": "2026-10-07T17:55:02.189604085Z",
      "endTime": "2026-10-07T18:19:08.046890686Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3aagkkq4l3ogcvf4h8o101v6q3m76 still exists (deletion dispatched at 2026-10-07T17:55:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xf53ccb2c090>: 
    resource group "rg-cordon-test-zfbgp6td9bvg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-zfbgp6td9bvg" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "name": "3df7278a-fc9c-4dad-bb05-8577c80b31f1",
      "status": "Failed",
      "startTime": "2026-10-07T17:55:02.189604085Z",
      "endTime": "2026-10-07T18:19:08.046890686Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3aagkkq4l3ogcvf4h8o101v6q3m76 still exists (deletion dispatched at 2026-10-07T17:55:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3e339eaf0eb8>: 
    resource group "autoscaling-cluster-wq59b44864g7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wq59b44864g7" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "name": "486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "status": "Failed",
      "startTime": "2026-10-07T17:54:04.264847355Z",
      "endTime": "2026-10-07T18:18:08.23090916Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta38okksp4b36cn28fmfj8oja6islo0 still exists (deletion dispatched at 2026-10-07T17:54:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3e339eaf0eb8>: 
    resource group "autoscaling-cluster-wq59b44864g7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wq59b44864g7" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "name": "486d4d3b-fd8e-4b61-92c0-db85f958748b",
      "status": "Failed",
      "startTime": "2026-10-07T17:54:04.264847355Z",
      "endTime": "2026-10-07T18:18:08.23090916Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta38okksp4b36cn28fmfj8oja6islo0 still exists (deletion dispatched at 2026-10-07T17:54:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19e0b121c138>: 
    resource group "admin-api-serialconsole-8wkg2h2hv8wb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-8wkg2h2hv8wb" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "name": "52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "status": "Failed",
      "startTime": "2026-10-07T17:56:40.94108576Z",
      "endTime": "2026-10-07T18:20:42.13673234Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cm8fj69efgottp7couqt5o1bu2r2 still exists (deletion dispatched at 2026-10-07T17:56:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19e0b121c138>: 
    resource group "admin-api-serialconsole-8wkg2h2hv8wb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-8wkg2h2hv8wb" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "name": "52bbbc8d-3e28-45ff-86eb-4bc453001256",
      "status": "Failed",
      "startTime": "2026-10-07T17:56:40.94108576Z",
      "endTime": "2026-10-07T18:20:42.13673234Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cm8fj69efgottp7couqt5o1bu2r2 still exists (deletion dispatched at 2026-10-07T17:56:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560

Command failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-<id>/pipeline@sha256:<hex> arohcpsvcdev.azurecr.io/ci-op-<id>/pipeline:<hex> --from-registry-confi...

full failure pattern:

Command failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-<id>/pipeline@sha256:<hex> arohcpsvcdev.azurecr.io/ci-op-<id>/pipeline:<hex> --from-registry-config /tmp/tmp.<id>/containers/auth.json --to-registry-config

Signal:

provision

none

Full failure examples (2)

time=2026-10-07T02:30:15.367Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> fleet.image.registry <nil>}:{<nil> fleet.image.repository <nil>}@{<nil> fleet.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-07T02:32:48.216Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.lx56KPpMDi/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-jhx30i3q/pipeline@sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 to arohcpsvcdev.azurecr.io/ci-op-jhx30i3q/pipeline:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604.\nThe image will still be available under it's original digest sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 in the target registry.\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unavailable: service unavailable: health check failed: please see /debug/health\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unavailable: service unavailable: health check failed: please see /debug/health\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-jhx30i3q/pipeline@sha256:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 arohcpsvcdev.azurecr.io/ci-op-jhx30i3q/pipeline:a606d40bf095b5c6399b022598c612994fd9b0a54dccde130e796be7b44b7604 --from-registry-config /tmp/tmp.lx56KPpMDi/containers/auth.json --to-registry-config /tmp/tmp.lx56KPpMDi/containers/auth.json\n exit status 1"

time=2026-10-07T02:34:43.899Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> kubeApplier.image.registry <nil>}:{<nil> kubeApplier.image.repository <nil>}@{<nil> kubeApplier.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-07T02:35:51.286Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.BaWVNiNbL6/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-n30v64rp/pipeline@sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 to arohcpsvcdev.azurecr.io/ci-op-n30v64rp/pipeline:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894.\nThe image will still be available under it's original digest sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 in the target registry.\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: recognizable error message not found: GET \"https://registry.build11.ci.openshift.org/v2/ci-op-n30v64rp/pipeline/manifests/sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894\": response status code 503: Service Unavailable\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-n30v64rp/pipeline@sha256:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 arohcpsvcdev.azurecr.io/ci-op-n30v64rp/pipeline:9703abd2b587ea12d843258574bb8f17d0cee4bd68cb43fcbd7116f6e22ea894 --from-registry-config /tmp/tmp.BaWVNiNbL6/containers/auth.json --to-registry-config /tmp/tmp.BaWVNiNbL6/containers/auth.json\n exit status 1"

Contributing tests (2)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Fleet/global/mirror-image	1
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.KubeApplier/global/mirror-image	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 01:41	PR #7105	2107647565049630720
2026-10-07 01:40	PR #7397	2107647281405628416

DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x213e82b661e0>: 
    resource group "rg-cluster-back-version-s9llrx7qtffw": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-s9llrx7qtffw", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x213e826ac820>{
                msg: "resource group \"rg-cluster-back-version-s9llrx7qtffw\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x213e826ac800>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x213e83e111d0>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x213e82b661e0>: 
    resource group "rg-cluster-back-version-s9llrx7qtffw": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-s9llrx7qtffw", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x213e826ac820>{
                msg: "resource group \"rg-cluster-back-version-s9llrx7qtffw\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x213e826ac800>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x213e83e111d0>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c28962ca078>: 
    resource group "rg-cluster-back-version-kcpxjfv4fpw8": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c289468e340>{
                msg: "resource group \"rg-cluster-back-version-kcpxjfv4fpw8\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x3c289468e320>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x3c2894a68660>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c28962ca078>: 
    resource group "rg-cluster-back-version-kcpxjfv4fpw8": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c289468e340>{
                msg: "resource group \"rg-cluster-back-version-kcpxjfv4fpw8\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x3c289468e320>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x3c2894a68660>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

Process did not finish before 15m0s timeout

full failure pattern:

Process did not finish before 15m0s timeout

Signal:

unknown

none

Full failure examples (2)

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-10-07T15:54:40Z"}

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-10-07T06:30:56Z"}

Contributing tests (1)

Failed At	Job	Test	Support
unknown	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Job run should complete before timeout	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

alert [hcp] HCPClusterOperatorUnavailable fired

full failure pattern:

alert [hcp] HCPClusterOperatorUnavailable fired

Signal:

alert

none

Full failure examples (2)

alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T14:28:55Z
  Ended:       2026-10-07T15:55:54Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtqjq0sahnttfr0obenao468e4qlb-e0e2j3m7x8c7o1b", region="westus3", severity="warning"
  Description: 10 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtqjq0sahnttfr0obenao468e4qlb-e0e2j3m7x8c7o1b (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:52Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vuf7r0mkidc4qe1opuvsfmq0jlia8-p9j3q3c5l3l9j9t", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vuf7r0mkidc4qe1opuvsfmq0jlia8-p9j3q3c5l3l9j9t (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:52Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vu5443vh0o9q88a97n0snmv6qq2pb-acr-pull", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vu5443vh0o9q88a97n0snmv6qq2pb-acr-pull (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:52Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vu0gdmpn5k5d5bgvk4606cnb2ql44-s5l1e0o2a6j6m8n", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vu0gdmpn5k5d5bgvk4606cnb2ql44-s5l1e0o2a6j6m8n (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 5:
  State:       Resolved
  Started:     2026-10-07T15:27:53Z
  Ended:       2026-10-07T15:46:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtnemv5o1grorom2pjgo9il7l5cgq-l7p8p3x8g9o9l5c", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtnemv5o1grorom2pjgo9il7l5cgq-l7p8p3x8g9o9l5c (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:27:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtt5ll44f8b35n9lm1o4n0oims33s-private-kas", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtt5ll44f8b35n9lm1o4n0oims33s-private-kas (management cluster ci01-j9974528-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 7:
  State:       Resolved
  Started:     2026-10-07T15:27:53Z
  Ended:       2026-10-07T15:47:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vtu5p75nk86f54m2uds6vreosd2p2-v0n4p7w8h2f0o9a", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vtu5p75nk86f54m2uds6vreosd2p2-v0n4p7w8h2f0o9a (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 8:
  State:       Resolved
  Started:     2026-10-07T15:27:53Z
  Ended:       2026-10-07T15:47:53Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j9974528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9vu3ea2tjsap1sokp49bjsntgc3llb-u9p1t5k3r2x5x0n", region="westus3", severity="warning"
  Description: 8 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9vu3ea2tjsap1sokp49bjsntgc3llb-u9p1t5k3r2x5x0n (management cluster ci01-j9974528-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T05:10:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlcah4jgmcp45v4eq4gnjhrn1t86e-ingress-cert-le", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlcah4jgmcp45v4eq4gnjhrn1t86e-ingress-cert-le (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 2:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:30:07Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlr9uea4a002u1vobs1eamhd9ij0g-l5p7g4y6r1u6h3x", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlr9uea4a002u1vobs1eamhd9ij0g-l5p7g4y6r1u6h3x (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 3:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:29:08Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nldv9i221aq0hikhv2urfttr6p4ag-j3v3v2v9p0l8t7b", region="westus3", severity="warning"
  Description: 8 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nldv9i221aq0hikhv2urfttr6p4ag-j3v3v2v9p0l8t7b (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-07T06:04:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlhakelm48kkcoo98ifgh0s48qjr9-x7k8b4r2l9a9q2i", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlhakelm48kkcoo98ifgh0s48qjr9-x7k8b4r2l9a9q2i (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 5:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:29:08Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nleqkohnk6u75anrikd0ihfs6jmk6-i8v1g1d3d9x0t2s", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nleqkohnk6u75anrikd0ihfs6jmk6-i8v1g1d3d9x0t2s (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 6:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:32:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nllqoph4r7lp0chrtmvsdfajj63mb-s0d9l7t2s9f2t9s", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nllqoph4r7lp0chrtmvsdfajj63mb-s0d9l7t2s9f2t9s (management cluster ci01-j2737280-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 7:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:30:07Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nldc2p59tvtq108i7fn6ktamnpfd9-private-kas", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nldc2p59tvtq108i7fn6ktamnpfd9-private-kas (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Firing 8:
  State:       Resolved
  Started:     2026-10-07T06:04:06Z
  Ended:       2026-10-07T06:33:06Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2737280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t9nlp25sdnfj4rsjsnnajlljlta2m49-acr-pull", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t9nlp25sdnfj4rsjsnnajlljlta2m49-acr-pull (management cluster ci01-j2737280-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

alert [svc] BackendControllerRetryHotLoop fired

full failure pattern:

alert [svc] BackendControllerRetryHotLoop fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T14:03:14Z
  Ended:       2026-10-07T14:20:15Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j9974528-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-07T04:40:19Z
  Ended:       2026-10-07T04:55:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j2737280-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

full failure pattern:

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x50103f9c9c0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-d5s257gp6qv6" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-d5s257gp6qv6\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x50103f9c9c0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-d5s257gp6qv6" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-d5s257gp6qv6\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x2c581ef3e4a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wcbt6xkttfqm" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-wcbt6xkttfqm\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0x2c581ef3e4a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wcbt6xkttfqm" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-wcbt6xkttfqm\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 14:32	PR #6954	2107841500657225728
2026-10-07 13:40	PR #6364	2107828301232869376

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x9d4c3cd8090>: 
    resource group "admin-api-breakglass-6smbnkdj4hzs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-6smbnkdj4hzs" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "name": "1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "status": "Failed",
      "startTime": "2026-10-07T17:53:56.233010929Z",
      "endTime": "2026-10-07T18:18:01.34812688Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cjpfhsndlhr2qta8lp3hig1f25is still exists (deletion dispatched at 2026-10-07T17:53:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x9d4c3cd8090>: 
    resource group "admin-api-breakglass-6smbnkdj4hzs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-6smbnkdj4hzs" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "name": "1ea14b34-3ced-4583-bf55-f2007e2d285f",
      "status": "Failed",
      "startTime": "2026-10-07T17:53:56.233010929Z",
      "endTime": "2026-10-07T18:18:01.34812688Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cjpfhsndlhr2qta8lp3hig1f25is still exists (deletion dispatched at 2026-10-07T17:53:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25222e7880f0>: 
    resource group "rg-zstream-upgrade-4-20-42mlb9pj8467": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-2ns5vg" in resourcegroup="rg-zstream-upgrade-4-20-42mlb9pj8467" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d",
      "name": "bb6159e1-d013-447b-ad27-c209bc94526d",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:52.899645021Z",
      "endTime": "2026-10-07T18:25:01.15966395Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebl56c239c6lvk3po879sfe0nus still exists (deletion dispatched at 2026-10-07T18:00:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25222e7880f0>: 
    resource group "rg-zstream-upgrade-4-20-42mlb9pj8467": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-2ns5vg" in resourcegroup="rg-zstream-upgrade-4-20-42mlb9pj8467" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb6159e1-d013-447b-ad27-c209bc94526d",
      "name": "bb6159e1-d013-447b-ad27-c209bc94526d",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:52.899645021Z",
      "endTime": "2026-10-07T18:25:01.15966395Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebl56c239c6lvk3po879sfe0nus still exists (deletion dispatched at 2026-10-07T18:00:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27dd6af64318>: 
    resource group "ingress-cert-le-h6vkqmnzj7lf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-h6vkqmnzj7lf" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "name": "6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "status": "Failed",
      "startTime": "2026-10-07T18:01:35.349390537Z",
      "endTime": "2026-10-07T18:25:41.362314296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebs5l8pbehlv64f21cvh9tk83jo still exists (deletion dispatched at 2026-10-07T18:01:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27dd6af64318>: 
    resource group "ingress-cert-le-h6vkqmnzj7lf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-h6vkqmnzj7lf" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "name": "6a648923-5c5a-40ca-8d8d-ae3a84c60046",
      "status": "Failed",
      "startTime": "2026-10-07T18:01:35.349390537Z",
      "endTime": "2026-10-07T18:25:41.362314296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3cebs5l8pbehlv64f21cvh9tk83jo still exists (deletion dispatched at 2026-10-07T18:01:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:47	PR #7400	2107875420446658560

alert [svc] ServiceMemoryDrift fired

full failure pattern:

alert [svc] ServiceMemoryDrift fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T15:26:41Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9974528-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jrr5z", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jrr5z (namespace arobit) on cluster ci01-j9974528-svc has averaged 124.3% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:59	PR #7407	2107818144989974528

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 9 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:27:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:27:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-07T23:34:27Z
  Ended:       2026-10-07T23:43:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta829bp1fhgrfupabf34hjhbao8sjia", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta829bp1fhgrfupabf34hjhbao8sjia/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-07T23:35:31Z
  Ended:       2026-10-07T23:45:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta82bca3t1j71tbsbphnabaj9385rs2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta82bca3t1j71tbsbphnabaj9385rs2/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-07T23:35:32Z
  Ended:       2026-10-07T23:54:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-07T23:59:27Z
  Ended:       2026-10-08T00:08:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8e00ategjs6jdt7dm9iouhjbvp29q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8e00ategjs6jdt7dm9iouhjbvp29q/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-08T00:00:27Z
  Ended:       2026-10-08T00:10:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8erefr60nd4djngqkit7pco8mj14u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8erefr60nd4djngqkit7pco8mj14u/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-08T00:00:27Z
  Ended:       2026-10-08T00:07:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8f7d8ssfok4nfel8v6oomstor9osq", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8f7d8ssfok4nfel8v6oomstor9osq/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-08T00:02:28Z
  Ended:       2026-10-08T00:14:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta8fb8j9edjipdiide2u4g57sv913hs", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ta8fb8j9edjipdiide2u4g57sv913hs/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 22:28	PR #7305	2107961335642853376

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x219764076460>: 
    DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x219764076440>{
            msg: "DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x219764076460>: 
    DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x219764076440>{
            msg: "DNS for agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2bp46.apps.aro.l8a4l4n9c3b5o6z.me9l.j6769152.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 08:33	PR #7397	2107751278686769152

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:37:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-07T23:45:38Z
  Ended:       2026-10-07T23:54:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2ta82divi7qc3dtot56dc9nrv2029osp/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T00:09:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2853376/providers/microsoft.monitor/accounts/services-j2853376", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-7t5pm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 22:28	PR #7305	2107961335642853376

ERROR: Please run 'az login' to setup account.

full failure pattern:

ERROR: Please run 'az login' to setup account.

Signal:

provision

none

Full failure examples (1)

time=2026-10-07T09:04:20.332Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 description="Step swift-vnet\n  Kind: Shell\n  Command: ./swift-vnet.sh\n"
time=2026-10-07T09:04:22.138Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: Launching container group swift-vnet-aks-net as globalMSI...\nERROR: Please run 'az login' to setup account.\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/swift-vnet (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 08:31	PR #7398	2107750698572582912

failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1 context deadline exceeded

full failure pattern:

failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-07T19:22:31.326Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events description="Step kube-events\n Kind: Helm\n"
time=2026-10-07T19:22:32.066Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events values="map[enabled:true image:map[digest:sha256:7c0c84fdd1dde7d32754d602c60e7432108ecef850d5bcd2efc549110fbda458 registry:kubernetesshared.azurecr.io repository:shared/kube-events] resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]]]"
time=2026-10-07T19:22:32.066Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-10-07T19:22:32.374Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events namespace=monitoring
time=2026-10-07T19:22:32.684Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events dryRun=false
time=2026-10-07T19:22:32.684Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-10-07T19:22:32.822Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-10-07T19:27:38.910Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-07T19:27:38.911Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/kube-events	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 18:42	PR #7361	2107904302923124736

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x1f1b762f1400>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-dzl9ch5fblqj" to finish creating: GET https://rp.j2675200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "name": "597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "status": "Failed",
      "startTime": "2026-10-07T17:41:10.05474314Z",
      "endTime": "2026-10-07T18:10:19.11453378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x1f1b762f1400>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-dzl9ch5fblqj" to finish creating: GET https://rp.j2675200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "name": "597912cd-0173-4a02-a3e6-da94ac3b74b6",
      "status": "Failed",
      "startTime": "2026-10-07T17:41:10.05474314Z",
      "endTime": "2026-10-07T18:10:19.11453378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:37	PR #7310	2107872947002675200

ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershi...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; hypershift NodePool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; hypershift NodePool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc0b67484b00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-b6bhhgvdr2ld" to finish updating: GET https://rp.j0967424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "name": "9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "status": "Failed",
      "startTime": "2026-10-07T18:15:51.4879876Z",
      "endTime": "2026-10-07T18:34:57.661146093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is 2, want 3; hypershift NodePool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc0b67484b00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-b6bhhgvdr2ld" to finish updating: GET https://rp.j0967424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "name": "9d8af708-d204-41c1-8bc5-4f60976f86b6",
      "status": "Failed",
      "startTime": "2026-10-07T18:15:51.4879876Z",
      "endTime": "2026-10-07T18:34:57.661146093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is 2, want 3; hypershift NodePool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine s0q3e0l7h6y3n3h-np-1-njcq9-gnl4f: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:48	PR #7322	2107875761300967424

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3ed0c46420d8>: 
    resource group "rg-negative-tests-z8rjxzl5wzq5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8rjxzl5wzq5" to finish deleting: GET https://rp.j8408192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "name": "92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "status": "Failed",
      "startTime": "2026-10-07T20:31:10.664805555Z",
      "endTime": "2026-10-07T20:55:12.515323239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3ed0c46420d8>: 
    resource group "rg-negative-tests-z8rjxzl5wzq5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8rjxzl5wzq5" to finish deleting: GET https://rp.j8408192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "name": "92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "status": "Failed",
      "startTime": "2026-10-07T20:31:10.664805555Z",
      "endTime": "2026-10-07T20:55:12.515323239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 18:41	PR #7288	2107904235558408192

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no mat...

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x1d103f80a620>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
      - pod pull-secret-test container pull-test: ContainerCreating ()
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        err: <*errors.errorString | 0x1d103f3186b0>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x1d103f80a620>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
      - pod pull-secret-test container pull-test: ContainerCreating ()
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        err: <*errors.errorString | 0x1d103f3186b0>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;\n  - pod pull-secret-test container pull-test: ContainerCreating ()",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 12:57	PR #7325	2107817608098091008

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T20:13:47

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0x4aa5231ef20>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);
    let _endTime = datetime(2026-10-07T20:13:47.1720653Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-gg7xds6svpzk'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T20:13:54.1282773Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77", ActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", SubActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityStack:"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x4aa53d27068>{
            errs: [
                <*fmt.wrapError | 0x4aa5231ef00>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x4aa5231eec0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x4aa52207ce0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x4aa52207ca0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x4aa52820300>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0x4aa5231ef20>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);
    let _endTime = datetime(2026-10-07T20:13:47.1720653Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-gg7xds6svpzk'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T20:13:54.1282773Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77", ActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", SubActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f024af6-e2e7-4ff7-91db-bc5319766d55", ActivityStack:"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x4aa53d27068>{
            errs: [
                <*fmt.wrapError | 0x4aa5231ef00>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x4aa5231eec0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x4aa52207ce0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x4aa52207ca0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T20:13:47.1720651Z);\nlet _endTime = datetime(2026-10-07T20:13:47.1720653Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-gg7xds6svpzk'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T20:13:54.1282773Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77\", ActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", SubActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f024af6-e2e7-4ff7-91db-bc5319766d55\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f1a232d6-957a-4fb2-b729-3a7bbce9fa77 ARID=9f024af6-e2e7-4ff7-91db-bc5319766d55 > GW.Http.CallContext/9f024af6-e2e7-4ff7-91db-bc5319766d55)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x4aa52820300>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 18:41	PR #7288	2107904235558408192