CIHealth Failure Patterns

Environment: DEV

Failure patterns: 40 · Failures matched: 111 · Total runs: 91 · Runs affected: 60

Failure pattern

Failed at

Trend

Also in

ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsof...

full failure pattern:

ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x14550d98c9a0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-p98fbdr6lfb6": PUT https://rp.j6887424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p98fbdr6lfb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x14550d98c9a0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-p98fbdr6lfb6": PUT https://rp.j6887424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p98fbdr6lfb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x1657ab71ce40>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-rkxd84qqqgg8": PUT https://rp.j9463680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rkxd84qqqgg8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x1657ab71ce40>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-rkxd84qqqgg8": PUT https://rp.j9463680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rkxd84qqqgg8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:00	PR #7397	2108316538086887424
2026-10-08 12:51	PR #7397	2108178486299463680

VerifyHyperShiftIngressCertificate timed out after 10m0

full failure pattern:

VerifyHyperShiftIngressCertificate timed out after 10m0

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.1	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 07:02	PR #7325	2108090608831696896
2026-10-08 02:14	PR #7325	2108018145892503552

context deadline exceeded. To troubleshoot, visit <url>

full failure pattern:

context deadline exceeded. To troubleshoot, visit <url>

Signal:

e2e

none

Full failure examples (1)

github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0x17a8dd744af0?, {0x7f91c02f24c0?, 0x17a8dd261560?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1075 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0x17a8dcfdf200, {0x7f91c02f24c0, 0x17a8dd261560}, {0xf9f89cb, 0x15}, {0x17a8dc60e129, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:508 +0x2a8
github.com/Azure/ARO-HCP/test/e2e.init.func34.2({0x7f91c02f24c0, 0x17a8dd261560})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:68 +0x139
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x101a3e60?, 0x17a8dd261560?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 134
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xdfb

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked: failed to get subscription ID: ClientSecretCredential: unable to resolve an endpoint: server response error:
 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0x17a8dd744af0?, {0x7f91c02f24c0?, 0x17a8dd261560?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1075 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0x17a8dcfdf200, {0x7f91c02f24c0, 0x17a8dd261560}, {0xf9f89cb, 0x15}, {0x17a8dc60e129, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:508 +0x2a8
github.com/Azure/ARO-HCP/test/e2e.init.func34.2({0x7f91c02f24c0, 0x17a8dd261560})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:68 +0x139
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x101a3e60?, 0x17a8dd261560?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 134
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xdfb

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked: failed to get subscription ID: ClientSecretCredential: unable to resolve an endpoint: server response error:
 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 16:13	PR #7232	2108229382379147264

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j4452224-mgmt-1","clusterName":"ci01-j445222

Signal:

provision

none

Full failure examples (1)

time=2026-10-08T14:32:31.059Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T15:02:30.832Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T14:32:31.068088679Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T14:32:32.982339672Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T14:32:33.61885433Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 13:42	PR #7427	2108191395574452224

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0x119e8d352a00>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);
    let _endTime = datetime(2026-10-08T14:29:33.7915314Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7dxbsk584tbw'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T14:29:42.5063608Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666", ActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", SubActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityType:"GW.Http.CallContext", ParentActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityStack:"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x119e8dac0f30>{
            errs: [
                <*fmt.wrapError | 0x119e8d3529e0>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x119e8d3529c0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x119e8d36b280>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x119e8d36b1a0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x119e8b843000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0x119e8d352a00>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);
    let _endTime = datetime(2026-10-08T14:29:33.7915314Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7dxbsk584tbw'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T14:29:42.5063608Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666", ActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", SubActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityType:"GW.Http.CallContext", ParentActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityStack:"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x119e8dac0f30>{
            errs: [
                <*fmt.wrapError | 0x119e8d3529e0>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x119e8d3529c0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x119e8d36b280>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x119e8d36b1a0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x119e8b843000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 13:01	PR #7387	2108181128597737472

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:11:31Z
  Ended:       2026-10-08T23:17:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0504704-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:16:25Z
  Ended:       2026-10-08T23:32:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3544576-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T19:18:13Z
  Ended:       2026-10-08T19:24:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8343424-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	19

Affected runs (19)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7389	2108317197410504704
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 18:00	PR #7414	2108256251258343424
2026-10-08 18:00	PR #7389	2108256213476052992
2026-10-08 16:13	PR #7232	2108229382379147264
2026-10-08 12:00	PR #7389	2108165661279653888
2026-10-08 11:50	PR #7401	2108163022403932160
2026-10-08 11:12	PR #7383	2108153534082977792
2026-10-08 11:09	PR #7364	2108152878592954368
2026-10-08 11:08	PR #6364	2108152449343688704
2026-10-08 09:41	PR #7288	2108130674367008768
2026-10-08 09:33	PR #7387	2108128762271895552
2026-10-08 09:12	PR #7420	2108123255498870784
2026-10-08 08:20	PR #7414	2108110245078241280
2026-10-08 04:44	PR #7414	2108055963964542976
2026-10-08 03:54	PR #7177	2108043334072143872
2026-10-08 03:07	PR #7412	2108031501605015552
2026-10-08 02:14	PR #7325	2108018145892503552
2026-10-08 02:04	PR #7232	2108015628659986432

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp"
Unexpected error:
    <*fmt.wrapError | 0x36065ae1e700>: 
    failed to create NodePool np-2b5cwp: failed waiting for nodepool="np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp" in resourcegroup="rg-candidate-5-1-mdbffxtcx8wr" to finish creating: GET https://rp.j6247936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "name": "e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "status": "Failed",
      "startTime": "2026-10-08T19:25:31.299024517Z",
      "endTime": "2026-10-08T19:54:32.257674658Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp"
Unexpected error:
    <*fmt.wrapError | 0x36065ae1e700>: 
    failed to create NodePool np-2b5cwp: failed waiting for nodepool="np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp" in resourcegroup="rg-candidate-5-1-mdbffxtcx8wr" to finish creating: GET https://rp.j6247936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "name": "e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "status": "Failed",
      "startTime": "2026-10-08T19:25:31.299024517Z",
      "endTime": "2026-10-08T19:54:32.257674658Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x1d9f3c92fc0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gcnbkpsxtjrv" to finish creating: GET https://rp.j5951872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "name": "511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "status": "Failed",
      "startTime": "2026-10-08T20:56:10.667754986Z",
      "endTime": "2026-10-08T21:25:14.560841698Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x1d9f3c92fc0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gcnbkpsxtjrv" to finish creating: GET https://rp.j5951872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "name": "511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "status": "Failed",
      "startTime": "2026-10-08T20:56:10.667754986Z",
      "endTime": "2026-10-08T21:25:14.560841698Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb"
Unexpected error:
    <*fmt.wrapError | 0x1e49d4d5a340>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb" in resourcegroup="rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj" to finish creating: GET https://rp.j0335104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646",
      "name": "b0463f50-61f7-4515-9352-022d3d794646",
      "status": "Failed",
      "startTime": "2026-10-08T16:41:51.052187772Z",
      "endTime": "2026-10-08T17:10:57.967451326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb"
Unexpected error:
    <*fmt.wrapError | 0x1e49d4d5a340>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb" in resourcegroup="rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj" to finish creating: GET https://rp.j0335104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646",
      "name": "b0463f50-61f7-4515-9352-022d3d794646",
      "status": "Failed",
      "startTime": "2026-10-08T16:41:51.052187772Z",
      "endTime": "2026-10-08T17:10:57.967451326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (9)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor	1

Affected runs (11)

Date (UTC)	Associated PR	Prow job
2026-10-08 18:12	PR #6364	2108259306926247936
2026-10-08 18:00	PR #7387	2108256216495951872
2026-10-08 15:37	PR #7431	2108220296300335104
2026-10-08 14:53	PR #7361	2108209233152446464
2026-10-08 12:00	PR #7389	2108165661279653888
2026-10-08 11:08	PR #6364	2108152449343688704
2026-10-08 09:37	PR #7158	2108129603208876032
2026-10-08 09:09	PR #7292	2108122723405271040
2026-10-08 08:16	PR #7389	2108109354774302720
2026-10-08 07:10	PR #7412	2108092712069632000
2026-10-08 03:54	PR #7177	2108043334072143872

alert [svc] KustoLogsDataStale fired

full failure pattern:

alert [svc] KustoLogsDataStale fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:22:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:23:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire	8

Affected runs (8)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7389	2108317197410504704
2026-10-08 22:02	PR #7364	2108317114874990592
2026-10-08 18:00	PR #7414	2108256251258343424
2026-10-08 17:29	PR #7371	2108248385097043968
2026-10-08 15:40	PR #7162	2108221120938250240
2026-10-08 13:01	PR #7387	2108181128597737472
2026-10-08 11:08	PR #6364	2108152449343688704
2026-10-08 09:41	PR #7288	2108130674367008768

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:04:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3153024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.30:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3153024/providers/microsoft.monitor/accounts/hcps-j3153024", namespace="ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-tbmch", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:04:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3153024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.30:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3153024/providers/microsoft.monitor/accounts/hcps-j3153024", namespace="ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-tbmch", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 6 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-sst2s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-sst2s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:58:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:59:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:07:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:07:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	7

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7292	2108317188833153024
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 18:12	PR #6364	2108259306926247936
2026-10-08 18:00	PR #7389	2108256213476052992
2026-10-08 14:06	PR #7422	2108197422965460992
2026-10-08 11:12	PR #7383	2108153534082977792
2026-10-08 04:44	PR #7414	2108055963964542976

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T21:07:52Z
  Ended:       2026-10-08T21:13:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5951872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tar6kfvsbfkgd4sfo2s5ak9up9mghnd-ephemeral-disk", pod="openshift-apiserver-f7bf46fd6-jsjg9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tar6kfvsbfkgd4sfo2s5ak9up9mghnd-ephemeral-disk/openshift-apiserver-f7bf46fd6-jsjg9 has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T18:18:15Z
  Ended:       2026-10-08T18:25:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9147264-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2taok4lfsu899t43cjah6nqtn60rbr6d-r8d4l6w3w4r4h6t", pod="openshift-apiserver-59b5875fdc-r9dd5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2taok4lfsu899t43cjah6nqtn60rbr6d-r8d4l6w3w4r4h6t/openshift-apiserver-59b5875fdc-r9dd5 has been in a non-ready state for longer than 5 minutes.

alert fired 57 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-gvdlf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-gvdlf has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-pj78n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-pj78n has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-jnjtd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-jnjtd has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-rplxb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-rplxb has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-htmxm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-htmxm has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-rv6cc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-rv6cc has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="cloud-network-config-controller-74769fccfb-k8kwp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/cloud-network-config-controller-74769fccfb-k8kwp has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-hgjs7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-hgjs7 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-gs5jc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-gs5jc has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-8chb5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-8chb5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-psbgs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-psbgs has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-4tbx4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-4tbx4 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-gjpp8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-gjpp8 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-j44kk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-j44kk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-mvbj5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-mvbj5 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-ss6rg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-ss6rg has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-snf5p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-snf5p has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-mdgsc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-mdgsc has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-cwkvx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-cwkvx has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-bnmrb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-bnmrb has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-cv4g6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-cv4g6 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-7n652", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-7n652 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-4hlqb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-4hlqb has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-w2spp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-w2spp has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-87cjv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-87cjv has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-ht9c9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-ht9c9 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-jwnrm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-jwnrm has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-8mmlh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-8mmlh has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-7pzw8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-7pzw8 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-672v2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-672v2 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-9c8996664-xqlcz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-9c8996664-xqlcz has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-f8gbx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-f8gbx has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-l29xs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-l29xs has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-mcs4j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-mcs4j has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-m8sbd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-m8sbd has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-fg868", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-fg868 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-d57l7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-d57l7 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-6qjzf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-6qjzf has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-ssj25", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-ssj25 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-2nvzx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-2nvzx has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-4rxvd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-4rxvd has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-l6dz6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-l6dz6 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-6qjld", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-6qjld has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-m9rtt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-m9rtt has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-2njx7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-2njx7 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-2t8lh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-2t8lh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-rhccp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-rhccp has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-g64ns", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-g64ns has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-5c9wj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-5c9wj has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-kdtjv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-kdtjv has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-5fld5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-5fld5 has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-skdmq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-skdmq has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-lfqpn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-lfqpn has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-kq58q", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-kq58q has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-pjnm6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-pjnm6 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-hztch", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-hztch has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-zgk6p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-zgk6p has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-08 18:00	PR #7387	2108256216495951872
2026-10-08 16:13	PR #7232	2108229382379147264
2026-10-08 03:07	PR #7412	2108031501605015552

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:01:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="cloud-network-config-controller-79b774bd6b-9pkrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6b4e41e1-2282-44a1-ab27-d2b7bffdbb7d"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller-79b774bd6b-9pkrz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:01:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="csi-snapshot-controller-7fcf9f696b-5dd5v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ade41fab-9ae3-4c1c-af6d-d25dc46782e7"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/csi-snapshot-controller-7fcf9f696b-5dd5v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="csi-snapshot-controller-7fcf9f696b-5dd5v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ade41fab-9ae3-4c1c-af6d-d25dc46782e7"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/csi-snapshot-controller-7fcf9f696b-5dd5v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="cloud-network-config-controller-79b774bd6b-9pkrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6b4e41e1-2282-44a1-ab27-d2b7bffdbb7d"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller-79b774bd6b-9pkrz (controller) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T06:00:31Z
  Ended:       2026-10-08T06:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9e83f444-711c-486c-b496-8152e0711540"
  Description: Pod ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-08T06:00:31Z
  Ended:       2026-10-08T06:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9e83f444-711c-486c-b496-8152e0711540"
  Description: Pod ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 04:44	PR #7414	2108055963964542976

alert [svc] MaestroAgentMetricsProxyDown fired

full failure pattern:

alert [svc] MaestroAgentMetricsProxyDown fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T15:04:42Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j1446656-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T15:05:40Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j1446656-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T14:57:45Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4452224-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 14:14	PR #7414	2108199393621446656
2026-10-08 13:42	PR #7427	2108191395574452224

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T05:54:34Z
  Ended:       2026-10-08T06:18:34Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-g29zr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-08T05:54:34Z
  Ended:       2026-10-08T06:19:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-g29zr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 04:44	PR #7414	2108055963964542976

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (1)

alert fired 11 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-2g4tl", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-2g4tl has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-l5mn9", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-l5mn9 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vfshb", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vfshb has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-qhgdx", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-qhgdx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-wm888", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-wm888 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-t9gmx", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-t9gmx has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-h7lkm", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-h7lkm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-w5s5h", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-w5s5h has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vwbm7", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vwbm7 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-n644b", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-n644b has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-gh9b9", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-gh9b9 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 18:11	PR #7433	2108258997998981120

clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>

full failure pattern:

clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x3dc0df3e260>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x3dc0df3e260>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 12:00	PR #7389	2108165661279653888

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xa118982a360>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dx54wv2l48hl" to finish creating: GET https://rp.j4542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "name": "54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "status": "Failed",
      "startTime": "2026-10-08T05:35:12.262324182Z",
      "endTime": "2026-10-08T06:04:17.39866058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xa118982a360>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dx54wv2l48hl" to finish creating: GET https://rp.j4542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "name": "54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "status": "Failed",
      "startTime": "2026-10-08T05:35:12.262324182Z",
      "endTime": "2026-10-08T06:04:17.39866058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 04:44	PR #7414	2108055963964542976

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng"
Unexpected error:
    <*fmt.wrapError | 0x1dd12c4c3a80>: 
    failed to create NodePool np-ft6nng: failed waiting for nodepool="np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng" in resourcegroup="rg-candidate-4-22-j25t2jm5j66l" to finish creating: GET https://rp.j9390848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng"
Unexpected error:
    <*fmt.wrapError | 0x1dd12c4c3a80>: 
    failed to create NodePool np-ft6nng: failed waiting for nodepool="np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng" in resourcegroup="rg-candidate-4-22-j25t2jm5j66l" to finish creating: GET https://rp.j9390848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 15:27	PR #7383	2108217820029390848

cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"

full failure pattern:

cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-7fcmrmvxxv7n/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-7fcmrmvxxv7n/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-ttr5gf9j2z4p/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-ttr5gf9j2z4p/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4tll47xvzcmb/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4tll47xvzcmb/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	7

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7414	2108317119643914240
2026-10-08 20:21	PR #7232	2108291662634553344
2026-10-08 18:00	PR #7180	2108256210653286400
2026-10-08 14:46	PR #7105	2108207385897078784
2026-10-08 08:16	PR #7389	2108109354774302720
2026-10-08 07:57	PR #6775	2108104508235059200
2026-10-08 04:44	PR #7414	2108055963964542976

alert [svc] LeaderElectionLeaseStale fired

full failure pattern:

alert [svc] LeaderElectionLeaseStale fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:07:32Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j5978496-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.233:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5978496/providers/microsoft.monitor/accounts/services-j5978496", namespace="klusterlet-2tapn8dlbj9p40f8ecef335ah1n44qgl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-snrs6", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tapn8dlbj9p40f8ecef335ah1n44qgl on cluster ci01-j5978496-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T18:47:49Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9147264-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9147264/providers/microsoft.monitor/accounts/services-j9147264", namespace="klusterlet-2taok2013r1ut7fvo0ulo5kc534ntq7s", pod="arohcp-monitor-kube-state-metrics-7988dc648c-fljb2", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2taok2013r1ut7fvo0ulo5kc534ntq7s on cluster ci01-j9147264-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:48:14Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9653888-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.155:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9653888/providers/microsoft.monitor/accounts/services-j9653888", namespace="klusterlet-2tak8n7sjj3mgsij1id8amijusdtem6a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-87ql4", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tak8n7sjj3mgsij1id8amijusdtem6a on cluster ci01-j9653888-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-10-08 18:00	PR #7361	2108256282895978496
2026-10-08 16:13	PR #7232	2108229382379147264
2026-10-08 12:00	PR #7389	2108165661279653888
2026-10-08 11:09	PR #7364	2108152878592954368
2026-10-08 07:45	PR #7378	2108101483491758080
2026-10-08 07:10	PR #7412	2108092712069632000

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1044cd538270>: 
    resource group "rg-hs-pre--l7mzsgckn5bq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-m4xdbw" in resourcegroup="rg-hs-pre--l7mzsgckn5bq" to finish deleting: GET https://rp.j8864256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33",
      "name": "81f1c7e5-bd06-48b6-8e00-126748584d33",
      "status": "Failed",
      "startTime": "2026-10-08T22:52:33.320294122Z",
      "endTime": "2026-10-08T23:16:35.834245337Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tasnkqnpkt06eq3lmg9dsk6e9purnb2 still exists (deletion dispatched at 2026-10-08T22:52:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1044cd538270>: 
    resource group "rg-hs-pre--l7mzsgckn5bq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-m4xdbw" in resourcegroup="rg-hs-pre--l7mzsgckn5bq" to finish deleting: GET https://rp.j8864256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33",
      "name": "81f1c7e5-bd06-48b6-8e00-126748584d33",
      "status": "Failed",
      "startTime": "2026-10-08T22:52:33.320294122Z",
      "endTime": "2026-10-08T23:16:35.834245337Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tasnkqnpkt06eq3lmg9dsk6e9purnb2 still exists (deletion dispatched at 2026-10-08T22:52:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3adaf74760a8>: 
    resource group "fips-enabled-khqp8vkktpzt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-khqp8vkktpzt" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "name": "407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "status": "Failed",
      "startTime": "2026-10-08T16:13:27.229044406Z",
      "endTime": "2026-10-08T16:37:32.626253468Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamus8q8drvpor1j4v73qnjn2pbcpag still exists (deletion dispatched at 2026-10-08T16:13:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3adaf74760a8>: 
    resource group "fips-enabled-khqp8vkktpzt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-khqp8vkktpzt" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "name": "407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "status": "Failed",
      "startTime": "2026-10-08T16:13:27.229044406Z",
      "endTime": "2026-10-08T16:37:32.626253468Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamus8q8drvpor1j4v73qnjn2pbcpag still exists (deletion dispatched at 2026-10-08T16:13:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1aa3a56380a8>: 
    resource group "private-kas-mmbkx8tpnrlv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-mmbkx8tpnrlv" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494",
      "name": "bb91148d-7554-4a55-b1dd-4854fa37b494",
      "status": "Failed",
      "startTime": "2026-10-08T16:22:19.235799645Z",
      "endTime": "2026-10-08T16:46:22.547532555Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamuoo5fqaudkk7q5op01b86l5utmqh still exists (deletion dispatched at 2026-10-08T16:22:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1aa3a56380a8>: 
    resource group "private-kas-mmbkx8tpnrlv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-mmbkx8tpnrlv" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494",
      "name": "bb91148d-7554-4a55-b1dd-4854fa37b494",
      "status": "Failed",
      "startTime": "2026-10-08T16:22:19.235799645Z",
      "endTime": "2026-10-08T16:46:22.547532555Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamuoo5fqaudkk7q5op01b86l5utmqh still exists (deletion dispatched at 2026-10-08T16:22:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-08 21:38	PR #6928	2108311064448864256
2026-10-08 14:53	PR #7361	2108209233152446464
2026-10-08 08:20	PR #7414	2108110245078241280

VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil

full failure pattern:

VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xed922aab58>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0xed9195f1e0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0xed91fd22c0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xed922aab58>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0xed9195f1e0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0xed91fd22c0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x7b70a422fc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x7b70a4e87a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x7b70a6fbd20>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x7b70a422fc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x7b70a4e87a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x7b70a6fbd20>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e29e0f51bc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e29e2b846a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e29e176e4f0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e29e0f51bc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e29e2b846a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e29e176e4f0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-08 16:11	PR #7325	2108228889548427264
2026-10-08 07:02	PR #7325	2108090608831696896
2026-10-08 02:14	PR #7325	2108018145892503552

ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement

full failure pattern:

ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement

Signal:

alert

none

Full failure examples (3)

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j7950976' could not be found."
  }
}
--------------------------------------------------------------------------------

Contributing tests (2)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert collection is complete	2
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert collection is complete	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 17:21	PR #7422	2108246398544646144
2026-10-08 12:16	PR #7226	2108169795277950976

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1c27e024540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-6zjcv2t4zcfg" to finish creating: GET https://rp.j0298368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "name": "7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "status": "Failed",
      "startTime": "2026-10-08T14:08:53.200082246Z",
      "endTime": "2026-10-08T14:38:00.180447244Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1c27e024540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-6zjcv2t4zcfg" to finish creating: GET https://rp.j0298368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "name": "7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "status": "Failed",
      "startTime": "2026-10-08T14:08:53.200082246Z",
      "endTime": "2026-10-08T14:38:00.180447244Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:444]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc4b0a06e918>: 
    failed to create NodePool agg-np-b: failed waiting for nodepool="agg-np-b" for cluster "agg-cluster" in resourcegroup="feature-aggregation-qnvb5555k8tb" to finish creating: GET https://rp.j8011648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "name": "7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "status": "Failed",
      "startTime": "2026-10-08T04:02:19.757714703Z",
      "endTime": "2026-10-08T04:31:24.163355482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:444]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc4b0a06e918>: 
    failed to create NodePool agg-np-b: failed waiting for nodepool="agg-np-b" for cluster "agg-cluster" in resourcegroup="feature-aggregation-qnvb5555k8tb" to finish creating: GET https://rp.j8011648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "name": "7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "status": "Failed",
      "startTime": "2026-10-08T04:02:19.757714703Z",
      "endTime": "2026-10-08T04:31:24.163355482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 13:08	PR #7403	2108182743610298368
2026-10-08 02:42	PR #7361	2108025089978011648

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2c4b80612378>: 
    resource group "ingress-cert-le-68vr9hbvh2st": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-68vr9hbvh2st" to finish deleting: GET https://rp.j3544576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050",
      "name": "ca06aaf0-429b-4724-9342-8885fc527050",
      "status": "Failed",
      "startTime": "2026-10-08T23:18:31.808957419Z",
      "endTime": "2026-10-08T23:42:33.737902461Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tat5kosr5birnmo003enu4ag9tvkblp still exists (deletion dispatched at 2026-10-08T23:18:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2c4b80612378>: 
    resource group "ingress-cert-le-68vr9hbvh2st": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-68vr9hbvh2st" to finish deleting: GET https://rp.j3544576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050",
      "name": "ca06aaf0-429b-4724-9342-8885fc527050",
      "status": "Failed",
      "startTime": "2026-10-08T23:18:31.808957419Z",
      "endTime": "2026-10-08T23:42:33.737902461Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tat5kosr5birnmo003enu4ag9tvkblp still exists (deletion dispatched at 2026-10-08T23:18:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126c34d780d8>: 
    resource group "rg-np-version-upgrade-d2kz66-n2vxth4prxxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-d2kz66" in resourcegroup="rg-np-version-upgrade-d2kz66-n2vxth4prxxw" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "name": "53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "status": "Failed",
      "startTime": "2026-10-08T17:58:56.699498743Z",
      "endTime": "2026-10-08T18:23:02.391989177Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tao6fb6en2d65csg2b2sr0ei58fvhru still exists (deletion dispatched at 2026-10-08T17:58:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126c34d780d8>: 
    resource group "rg-np-version-upgrade-d2kz66-n2vxth4prxxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-d2kz66" in resourcegroup="rg-np-version-upgrade-d2kz66-n2vxth4prxxw" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "name": "53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "status": "Failed",
      "startTime": "2026-10-08T17:58:56.699498743Z",
      "endTime": "2026-10-08T18:23:02.391989177Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tao6fb6en2d65csg2b2sr0ei58fvhru still exists (deletion dispatched at 2026-10-08T17:58:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 16:11	PR #7325	2108228889548427264

ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights

full failure pattern:

ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights

Signal:

alert

none

Full failure examples (2)

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j7950976' could not be found."
  }
}
--------------------------------------------------------------------------------

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [infra] alert collection is complete	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 17:21	PR #7422	2108246398544646144
2026-10-08 12:16	PR #7226	2108169795277950976

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (2)

time=2026-10-08T16:19:05.365Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T16:30:17.342Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T16:19:05.375275155Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T16:19:08.114395254Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T16:19:08.462375095Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-08T16:30:17.335389178Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6F2FEEE9F3FAC221\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j5048320-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:19:14.8133695Z\",\"duration\":\"PT9M5.9123598S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:30:17.335532727Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"98034B352C012EE7\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j5048320-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT4.7536001S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:30:17.33558093Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DE129F3B0C485FAF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j5048320-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT1M38.1712414S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:30:17.335599002Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B325BEA3E7A07E7A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j5048320-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT1M56.0129258S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:30:17.335620623Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"D7A0CEADFD719B73\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j5048320-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT3.9567105S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:30:17.335657835Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7F61B179B809D260\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1\",\"resourceName\":\"ci01-j5048320-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:19:14.8133695Z\",\"duration\":\"PT9M0.9591654S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:30:17.335709239Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584101305363180932\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"11m11.960448525s\"}\n exit status 1"

time=2026-10-08T16:14:31.581Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T16:22:39.946Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T16:14:31.590606721Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T16:14:33.50353792Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T16:14:33.891298669Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-08T16:22:39.944584337Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"241855BF9A0D266F\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j3418880-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT1M38.1681956S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:22:39.944653829Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5B81E75ADA721336\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j3418880-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT4.9210349S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:22:39.94468287Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"0A9A6CB955E8A5E9\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j3418880-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:14:38.0622167Z\",\"duration\":\"PT7M50.9531763S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:22:39.944693871Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"127F3C3755EDC782\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j3418880-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT4.2491544S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:22:39.944712361Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1AF780C01A639DF0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j3418880-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT4.2335315S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:22:39.944730542Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2DBAB0983C574AF5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1\",\"resourceName\":\"ci01-j3418880-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:14:38.0622167Z\",\"duration\":\"PT5M54.4662985S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:22:39.944752483Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584101308111130568\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"8m8.354161621s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-08 15:54	PR #6928	2108224410405048320
2026-10-08 15:47	PR #7364	2108222712823418880

alert [svc] FrontendPathLatency fired

full failure pattern:

alert [svc] FrontendPathLatency fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:31:34Z
  Ended:       2026-10-09T00:04:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4990592-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j4990592-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7364	2108317114874990592

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2e1efc3180c0>: 
    resource group "admin-api-serialconsole-bootdiag-2vphmzvmmzbh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-2vphmzvmmzbh" to finish deleting: GET https://rp.j4279424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "name": "7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:07.770334262Z",
      "endTime": "2026-10-08T16:41:10.588488129Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tan1stnb9qf5nfuks8bh5mvhk8g2eul still exists (deletion dispatched at 2026-10-08T16:17:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2e1efc3180c0>: 
    resource group "admin-api-serialconsole-bootdiag-2vphmzvmmzbh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-2vphmzvmmzbh" to finish deleting: GET https://rp.j4279424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "name": "7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:07.770334262Z",
      "endTime": "2026-10-08T16:41:10.588488129Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tan1stnb9qf5nfuks8bh5mvhk8g2eul still exists (deletion dispatched at 2026-10-08T16:17:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 15:15	PR #7305	2108214686754279424

ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x18aecfdf0aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dq2w5s6mnvnb" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "name": "5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "status": "Failed",
      "startTime": "2026-10-08T08:17:53.86568203Z",
      "endTime": "2026-10-08T08:18:02.050800903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x18aecfdf0aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dq2w5s6mnvnb" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "name": "5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "status": "Failed",
      "startTime": "2026-10-08T08:17:53.86568203Z",
      "endTime": "2026-10-08T08:18:02.050800903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 07:10	PR #7412	2108092712069632000

ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/user...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/velero' was not found.; provider Microsoft.ManagedIdentity

Signal:

provision

none

Full failure examples (1)

time=2026-10-08T03:34:00.150Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-10-08T03:34:00.987Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-10-08T03:34:02.023Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=058182c57c37b78b09387fdf90aeaea50b1d526f842d6127dc4d5ebd76617220 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9563392-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F058182c57c37b78b09387fdf90aeaea50b1d526f842d6127dc4d5ebd76617220
time=2026-10-08T03:36:02.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.Resources/deployments/058182c57c37b78b09387fdf90aeaea50b1d526f842d6127dc4d5ebd76617220/operationStatuses/08584101764439358601\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-post\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"NotFound\\\",\\r\\n            \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/velero' was not found.\\\"\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"NotFound\\\",\\r\\n            \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mgmt-agent' was not found.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"NotFound\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"NotFound\\\",\\r\\n    \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/velero' was not found.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 02:42	PR #7414	2108025152309563392

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1ae233ff2078>: 
    resource group "v20260901preview-hqg6wzs5tgtz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-hqg6wzs5tgtz" to finish deleting: GET https://rp.j2977792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951",
      "name": "b7f165f5-05d4-46b6-9b64-0134316e8951",
      "status": "Failed",
      "startTime": "2026-10-08T12:28:19.710418085Z",
      "endTime": "2026-10-08T12:52:26.28519628Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tajia53q41l8grpaq7k8eu3qvn81ojn still exists (deletion dispatched at 2026-10-08T12:28:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1ae233ff2078>: 
    resource group "v20260901preview-hqg6wzs5tgtz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-hqg6wzs5tgtz" to finish deleting: GET https://rp.j2977792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951",
      "name": "b7f165f5-05d4-46b6-9b64-0134316e8951",
      "status": "Failed",
      "startTime": "2026-10-08T12:28:19.710418085Z",
      "endTime": "2026-10-08T12:52:26.28519628Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tajia53q41l8grpaq7k8eu3qvn81ojn still exists (deletion dispatched at 2026-10-08T12:28:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 11:12	PR #7383	2108153534082977792

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xd2898b7c180>: 
    resource group "private-kas-f55dwxz4lh8f": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-f55dwxz4lh8f" to finish deleting: GET https://rp.j8241280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "name": "f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "status": "Failed",
      "startTime": "2026-10-08T10:06:03.074876755Z",
      "endTime": "2026-10-08T10:30:10.535297041Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tahe6f6jvv1r5au11pgmqub5ah0te26 still exists (deletion dispatched at 2026-10-08T10:06:03Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xd2898b7c180>: 
    resource group "private-kas-f55dwxz4lh8f": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-f55dwxz4lh8f" to finish deleting: GET https://rp.j8241280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "name": "f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "status": "Failed",
      "startTime": "2026-10-08T10:06:03.074876755Z",
      "endTime": "2026-10-08T10:30:10.535297041Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tahe6f6jvv1r5au11pgmqub5ah0te26 still exists (deletion dispatched at 2026-10-08T10:06:03Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 08:20	PR #7414	2108110245078241280

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x44478a7c150>: 
    resource group "arm64-vm-cluster-ldnw4gs8dwrw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ldnw4gs8dwrw" to finish deleting: GET https://rp.j7078784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1",
      "name": "fcf937e7-61c6-4778-aabd-893fd37924c1",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:57.9406496Z",
      "endTime": "2026-10-08T16:41:57.233664583Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x44478a7c150>: 
    resource group "arm64-vm-cluster-ldnw4gs8dwrw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ldnw4gs8dwrw" to finish deleting: GET https://rp.j7078784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1",
      "name": "fcf937e7-61c6-4778-aabd-893fd37924c1",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:57.9406496Z",
      "endTime": "2026-10-08T16:41:57.233664583Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 14:46	PR #7105	2108207385897078784

VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-<id> in namespace e2e-fips-test-<id> on node <node> after retries: an...

full failure pattern:

VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-<id> in namespace e2e-fips-test-<id> on node <node> after retries: an error on the server ("unknown") has prevented the request from succeeding (get

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0x3e609f8540a8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-9c9j4)
    {
        errs: [
            <*fmt.wrapError | 0x3e609fd10180>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                err: <*fmt.wrapError | 0x3e609fd10160>{
                    msg: "failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                    err: <*fmt.wrapError | 0x3e609fd10020>{
                        msg: "failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                        err: <*errors.StatusError | 0x3e609eee2c80>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-9c9j4",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0x3e609f8540a8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-9c9j4)
    {
        errs: [
            <*fmt.wrapError | 0x3e609fd10180>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                err: <*fmt.wrapError | 0x3e609fd10160>{
                    msg: "failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                    err: <*fmt.wrapError | 0x3e609fd10020>{
                        msg: "failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                        err: <*errors.StatusError | 0x3e609eee2c80>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-9c9j4",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 02:42	PR #7288	2108025105396273152

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x1409e4db1080>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1409e4324dc0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1409e4f4c060>{
                    msg: "DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1409e4f4c040>{
                        msg: "DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x1409e4db1080>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1409e4324dc0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1409e4f4c060>{
                    msg: "DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1409e4f4c040>{
                        msg: "DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 07:57	PR #6775	2108104508235059200

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x255cb079f740>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dgrj4944q7jq" to finish creating: GET https://rp.j5460992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "name": "5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "status": "Failed",
      "startTime": "2026-10-08T15:30:05.316469426Z",
      "endTime": "2026-10-08T15:59:10.872777291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x255cb079f740>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dgrj4944q7jq" to finish creating: GET https://rp.j5460992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "name": "5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "status": "Failed",
      "startTime": "2026-10-08T15:30:05.316469426Z",
      "endTime": "2026-10-08T15:59:10.872777291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 14:06	PR #7422	2108197422965460992

failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1

full failure pattern:

failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1

Signal:

provision

none

Full failure examples (1)

time=2026-10-08T03:31:09.066Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n"
time=2026-10-08T03:31:09.688Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpci01 kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:c5d026c4a8335c2d098ce04776e25babc3e7773e70817b5b709f388bb52488d2] metricLabelsAllowlist:[nodes=[node.kubernetes.io/instance-type,kubernetes.azure.com/agentpool]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[system]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:9c285c83d939df7d0bdf27287562bb6fcf8fd6ab639d899abba1f14a0c2d5647] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:5fc5d09d0245cc1e7c6432ca9b0ebcc652c047b550d0289ea45725c14343585d]] thanosImage:map[registry:arohcpsvcdev.azurecr.io repository:thanos/thanos sha:b567818fe608067eb0f1d7c2c4fe361e7ad83c8a256234c97685f1d0bf670cc8 tag:v0.41.0] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[serviceAccount:map[managedIdentity:0b00177b-a54e-4160-b95f-d8ba55b89f42]] prometheusSpec:map[externalLabels:map[cluster:ci01-j0275840-svc environment:ci01 region:westus3] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:7ef9c1d80e4e8e1eec1a207166af22930c8f11200f25001d1a3285b30b4f13f9] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-ci01-j0275840-svc-ng1v.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-d00da46377ca403ab27fbe4e13d51efe/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 replicas:2 resources:map[limits:map[memory:NONE] requests:map[cpu:NONE memory:NONE]] shards:1 version: zoneCount:3] schedulingRole:system]"
time=2026-10-08T03:31:09.688Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-10-08T03:31:10.214Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus
time=2026-10-08T03:31:10.584Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false
time=2026-10-08T03:31:10.584Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-10-08T03:31:10.783Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-10-08T03:40:20.024Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-08T03:40:20.024Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 02:48	PR #6775	2108026812570275840

failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded

full failure pattern:

failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-08T14:39:49.369Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-08T14:39:51.193Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-08T14:39:53.050Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4570d5430c346685aef785faa645617cca8a23038712be321dc348c5ac428645 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1446656-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4570d5430c346685aef785faa645617cca8a23038712be321dc348c5ac428645
time=2026-10-08T15:09:49.053Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 14:14	PR #7414	2108199393621446656

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResourc...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1c054b45c180>: 
    resource group "rg-candidate-4-21-l7wm67p4w7mm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-pjnsh8" in resourcegroup="rg-candidate-4-21-l7wm67p4w7mm" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac",
      "name": "de22b374-9b97-455f-b396-0c997a7b79ac",
      "status": "Failed",
      "startTime": "2026-10-08T17:28:41.684717507Z",
      "endTime": "2026-10-08T17:52:48.530098677Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1c054b45c180>: 
    resource group "rg-candidate-4-21-l7wm67p4w7mm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-pjnsh8" in resourcegroup="rg-candidate-4-21-l7wm67p4w7mm" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac",
      "name": "de22b374-9b97-455f-b396-0c997a7b79ac",
      "status": "Failed",
      "startTime": "2026-10-08T17:28:41.684717507Z",
      "endTime": "2026-10-08T17:52:48.530098677Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 16:11	PR #7325	2108228889548427264

console should return a successful response or redirect, got 503

full failure pattern:

console should return a successful response or redirect, got 503

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:312]: Timed out after 2700.006s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:310 with:
console should return a successful response or redirect, got 503
Expected
    <int>: 503
to be <
    <int>: 400
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:312]: Timed out after 2700.006s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:310 with:
console should return a successful response or redirect, got 503
Expected
    <int>: 503
to be <
    <int>: 400

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 13:35	PR #6807	2108189567159898112