CIHealth Failure Patterns

Environment: DEV

Failure patterns: 28 · Failures matched: 44 · Total runs: 48 · Runs affected: 21

Failure pattern

Failed at

Trend

Also in

VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39="Pa...

full failure pattern:

VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39="Partial"; Failing=True: Cluster operator console is not available

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:122]: cluster "cluster-hs-pre-546c5c" should complete its version rollout
Unexpected error:
    <*fmt.wrapError | 0x2e4bde720b40>: 
    VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39="Partial"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available
    {
        msg: "VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
        err: <*verifiers.pollObservationError | 0x2e4bde720ac0>{
            observation: <*errors.errorString | 0x2e4be03290f0>{
                s: "clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
            },
            interruption: <*fmt.wrapError | 0x2e4bde720aa0>{
                msg: "failed to get clusterversion \"version\": context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:122]: cluster "cluster-hs-pre-546c5c" should complete its version rollout
Unexpected error:
    <*fmt.wrapError | 0x2e4bde720b40>: 
    VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39="Partial"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available
    {
        msg: "VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
        err: <*verifiers.pollObservationError | 0x2e4bde720ac0>{
            observation: <*errors.errorString | 0x2e4be03290f0>{
                s: "clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
            },
            interruption: <*fmt.wrapError | 0x2e4bde720aa0>{
                msg: "failed to get clusterversion \"version\": context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 05:28	PR #7727	2114590084358475776

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-<id>: expected

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1623311283c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x1623329ba440>{
            s: "dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1623311283c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x1623329ba440>{
            s: "dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 03:00	PR #7615	2114552936569769984

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster ->

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2148d2b87580>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x2148d2b87560>{
            observation: <*errors.errorString | 0x2148d2b63740>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x2148d2b87540>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2148d2b84b90>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2148d2b87500>{
                        msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2148d332b860>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2148d2b87580>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x2148d2b87560>{
            observation: <*errors.errorString | 0x2148d2b63740>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x2148d2b87540>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2148d2b84b90>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2148d2b87500>{
                        msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2148d332b860>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 20:54	PR #7177	2114823017199570944

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-<id>: expected all

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x22cffa244a80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x22cffa418620>{
            s: "dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x22cffa244a80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x22cffa418620>{
            s: "dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 12:47	PR #7757	2114700407203696640

at least one schedule should have a LastBackupTime

full failure pattern:

at least one schedule should have a LastBackupTime

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.006s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.006s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 23:28	PR #7766	2114861755761954816

failed to create service principal: create service principal for appId "<uuid>": Post "<url>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

full failure pattern:

failed to create service principal: create service principal for appId "<uuid>": Post "<url>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:131]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0x1b303bad2a00>: 
    failed to create service principal: create service principal for appId "a52f84ec-ecc1-43ca-9731-aac8f9f81ad5": Post "https://graph.microsoft.com/v1.0/servicePrincipals": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    {
        msg: "failed to create service principal: create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
        err: <*fmt.wrapError | 0x1b303bad2980>{
            msg: "create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
            err: <*url.Error | 0x1b303c6b42d0>{
                Op: "Post",
                URL: "https://graph.microsoft.com/v1.0/servicePrincipals",
                Err: <*http.timeoutError | 0x1b303d466f50>{
                    err: "net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:131]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0x1b303bad2a00>: 
    failed to create service principal: create service principal for appId "a52f84ec-ecc1-43ca-9731-aac8f9f81ad5": Post "https://graph.microsoft.com/v1.0/servicePrincipals": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    {
        msg: "failed to create service principal: create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
        err: <*fmt.wrapError | 0x1b303bad2980>{
            msg: "create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
            err: <*url.Error | 0x1b303c6b42d0>{
                Op: "Post",
                URL: "https://graph.microsoft.com/v1.0/servicePrincipals",
                Err: <*http.timeoutError | 0x1b303d466f50>{
                    err: "net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 10:29	PR #7665	2114665780556599296

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xe6261437860>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xe6261437860>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 07:00	PR #7662	2114613195741073408

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x3952e9095e40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x3952e9095e40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 14:55	PR #7662	2114732780888788992

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 603.481s.
Expected success, but got an error:
    <*errors.errorString | 0x12a5a347cf40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 603.481s.
Expected success, but got an error:
    <*errors.errorString | 0x12a5a347cf40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 09:27	PR #7712	2114650067662016512

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.003s.
Expected success, but got an error:
    <*errors.errorString | 0x294c3a4e6f50>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.003s.
Expected success, but got an error:
    <*errors.errorString | 0x294c3a4e6f50>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 05:30	PR #7712	2114590452421234688

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T00:54:17Z
  Ended:       2026-10-27T01:00:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1954816-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tmpuqe3u54c0c1kt3dhd472afk621tp-agg-cluster", pod="ignition-server-proxy-cccb6f6bd-9jw8t", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tmpuqe3u54c0c1kt3dhd472afk621tp-agg-cluster/ignition-server-proxy-cccb6f6bd-9jw8t has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T16:08:39Z
  Ended:       2026-10-26T16:14:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8788992-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tmic1o03pdh91igeu5gq690e02l56kv-private-kas", pod="kube-scheduler-84685ff5d-cz6cm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tmic1o03pdh91igeu5gq690e02l56kv-private-kas/kube-scheduler-84685ff5d-cz6cm has been in a non-ready state for longer than 5 minutes.

alert fired 8 time(s) (1 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:34:53Z
  Ended:       2026-10-26T15:40:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7915648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tmhn1j616q7r7qo51i9d3pca09c16db-h2y9l6v1q7f6h4e", pod="cluster-network-operator-5f67bbdd9d-4fb4z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tmhn1j616q7r7qo51i9d3pca09c16db-h2y9l6v1q7f6h4e/cluster-network-operator-5f67bbdd9d-4fb4z has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-26 23:28	PR #7766	2114861755761954816
2026-10-26 14:55	PR #7662	2114732780888788992
2026-10-26 13:47	PR #7758	2114715638197915648
2026-10-26 02:50	PR #7721	2114550200386195456

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:18:52Z
  Ended:       2026-10-26T15:45:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9290d46d-5d55-4654-a54f-7d8738f6108c"
  Description: Pod ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-26T15:19:52Z
  Ended:       2026-10-26T15:45:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9290d46d-5d55-4654-a54f-7d8738f6108c"
  Description: Pod ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T06:47:16Z
  Ended:       2026-10-26T07:14:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5b32bab-4c53-46e1-80dd-3f2bfd84e7a7"
  Description: Pod ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-26T06:47:16Z
  Ended:       2026-10-26T07:14:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5b32bab-4c53-46e1-80dd-3f2bfd84e7a7"
  Description: Pod ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 9 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T04:06:15Z
  Ended:       2026-10-26T04:15:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cloud-controller-manager", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g", pod="azure-cloud-controller-manager-8447d8b694-4zr5b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d02104f0-060b-4edb-8a6c-e45f411809b0"
  Description: Pod ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g/azure-cloud-controller-manager-8447d8b694-4zr5b (cloud-controller-manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-26T04:06:16Z
  Ended:       2026-10-26T04:15:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cloud-controller-manager", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g", pod="azure-cloud-controller-manager-8447d8b694-4zr5b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d02104f0-060b-4edb-8a6c-e45f411809b0"
  Description: Pod ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g/azure-cloud-controller-manager-8447d8b694-4zr5b (cloud-controller-manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:15:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7padnde7vvi5gn0v8i8h43umncmvl-cilium-cl", pod="csi-snapshot-controller-84dc6bbd7d-68jr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07c48f71-4da8-4382-849e-8180c459ebb1"
  Description: Pod ocm-arohcpci01-2tm7padnde7vvi5gn0v8i8h43umncmvl-cilium-cl/csi-snapshot-controller-84dc6bbd7d-68jr9 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="multus-admission-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="multus-admission-controller-6f7d476866-zcg5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8fa3adbb-8673-43f6-8f2c-cb4b4dc5a2ff"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/multus-admission-controller-6f7d476866-zcg5s (multus-admission-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:18:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cluster-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="cluster-policy-controller-7c486c777d-94qnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d0f7a448-44aa-43fc-891a-29f853013cc6"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/cluster-policy-controller-7c486c777d-94qnj (cluster-policy-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="multus-admission-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="multus-admission-controller-6f7d476866-zcg5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8fa3adbb-8673-43f6-8f2c-cb4b4dc5a2ff"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/multus-admission-controller-6f7d476866-zcg5s (multus-admission-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:18:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cluster-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="cluster-policy-controller-7c486c777d-94qnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d0f7a448-44aa-43fc-891a-29f853013cc6"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/cluster-policy-controller-7c486c777d-94qnj (cluster-policy-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Resolved
  Started:     2026-10-26T04:09:15Z
  Ended:       2026-10-26T04:20:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="csi-snapshot-controller-56f74747dd-q2c9t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3dba828e-0851-4048-b70e-3efa4c0ac3c5"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/csi-snapshot-controller-56f74747dd-q2c9t (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Resolved
  Started:     2026-10-26T04:09:15Z
  Ended:       2026-10-26T04:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="csi-snapshot-controller-56f74747dd-q2c9t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3dba828e-0851-4048-b70e-3efa4c0ac3c5"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/csi-snapshot-controller-56f74747dd-q2c9t (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-26 13:47	PR #7758	2114715638197915648
2026-10-26 05:27	PR #7739	2114589729923010560
2026-10-26 02:50	PR #7721	2114550200386195456

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:30:55Z
  Ended:       2026-10-26T15:41:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T15:30:55Z
  Ended:       2026-10-26T15:41:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T06:59:16Z
  Ended:       2026-10-26T07:10:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T06:59:17Z
  Ended:       2026-10-26T07:10:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-26 13:47	PR #7758	2114715638197915648
2026-10-26 05:27	PR #7739	2114589729923010560

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-rd6g78" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2215eed744c0>: 
    failed to create HCP cluster cluster-zstream-4-23-rd6g78: failed waiting for cluster="cluster-zstream-4-23-rd6g78" in resourcegroup="rg-zstream-upgrade-4-23-87rsp94v6tm5" to finish creating: GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "name": "aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "status": "Failed",
      "startTime": "2026-10-26T14:55:15.93882507Z",
      "endTime": "2026-10-26T15:24:15.95371054Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
            correlation ID: b0d46297-d5fe-44e3-bdff-447f2228b7af
            client request ID: fb0cf278-59cc-4ae8-8936-61b404d6c765
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-rd6g78" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2215eed744c0>: 
    failed to create HCP cluster cluster-zstream-4-23-rd6g78: failed waiting for cluster="cluster-zstream-4-23-rd6g78" in resourcegroup="rg-zstream-upgrade-4-23-87rsp94v6tm5" to finish creating: GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "name": "aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "status": "Failed",
      "startTime": "2026-10-26T14:55:15.93882507Z",
      "endTime": "2026-10-26T15:24:15.95371054Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
            correlation ID: b0d46297-d5fe-44e3-bdff-447f2228b7af
            client request ID: fb0cf278-59cc-4ae8-8936-61b404d6c765
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x3625ebf8a680>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gnkr42cnrl9g" to finish creating: GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549",
      "name": "c3982b99-da94-4694-b894-3515f5f00549",
      "status": "Failed",
      "startTime": "2026-10-26T06:24:09.477795034Z",
      "endTime": "2026-10-26T06:53:17.186342249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
            correlation ID: 9f2b05a4-0a2f-41bc-8784-edb61a547f4b
            client request ID: 3e8ce286-5388-4234-a7ff-6e068949d596
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x3625ebf8a680>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gnkr42cnrl9g" to finish creating: GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549",
      "name": "c3982b99-da94-4694-b894-3515f5f00549",
      "status": "Failed",
      "startTime": "2026-10-26T06:24:09.477795034Z",
      "endTime": "2026-10-26T06:53:17.186342249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
            correlation ID: 9f2b05a4-0a2f-41bc-8784-edb61a547f4b
            client request ID: 3e8ce286-5388-4234-a7ff-6e068949d596
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-26 13:47	PR #7758	2114715638197915648
2026-10-26 05:27	PR #7739	2114589729923010560

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2d970cc94090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
      "name": "c8761660-38f6-4612-a201-a6ddec925f84",
      "status": "Failed",
      "startTime": "2026-10-26T09:38:13.907922394Z",
      "endTime": "2026-10-26T09:57:20.458727782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
            correlation ID: 4c993536-8323-4301-a24e-f6918a189628
            client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2d970cc94090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
      "name": "c8761660-38f6-4612-a201-a6ddec925f84",
      "status": "Failed",
      "startTime": "2026-10-26T09:38:13.907922394Z",
      "endTime": "2026-10-26T09:57:20.458727782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
            correlation ID: 4c993536-8323-4301-a24e-f6918a189628
            client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x13f2532896c0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "status": "Failed",
      "startTime": "2026-10-26T03:58:11.830961438Z",
      "endTime": "2026-10-26T04:17:16.025648143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
            correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
            client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x13f2532896c0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "status": "Failed",
      "startTime": "2026-10-26T03:58:11.830961438Z",
      "endTime": "2026-10-26T04:17:16.025648143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
            correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
            client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-26 08:29	PR #7573	2114635485509849088
2026-10-26 03:00	PR #7615	2114552936569769984

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:16:58Z
  Ended:       2026-10-26T15:41:54Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T15:16:58Z
  Ended:       2026-10-26T15:41:55Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T06:44:17Z
  Ended:       2026-10-26T07:09:15Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T06:45:17Z
  Ended:       2026-10-26T07:14:20Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-26 13:47	PR #7758	2114715638197915648
2026-10-26 05:27	PR #7739	2114589729923010560

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x75e04e9f9a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-lmxnc7" in resourcegroup="rg-np-version-upgrade-lmxnc7-q9mf48fq597k" to finish creating: GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "name": "d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "status": "Failed",
      "startTime": "2026-10-26T16:25:47.925761101Z",
      "endTime": "2026-10-26T16:44:52.193515957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
            correlation ID: f7b1fc07-103a-435d-9acb-75f17e0334fb
            client request ID: 9e0fce66-a04c-49ec-96a9-4c1b9377e626
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x75e04e9f9a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-lmxnc7" in resourcegroup="rg-np-version-upgrade-lmxnc7-q9mf48fq597k" to finish creating: GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "name": "d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "status": "Failed",
      "startTime": "2026-10-26T16:25:47.925761101Z",
      "endTime": "2026-10-26T16:44:52.193515957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
            correlation ID: f7b1fc07-103a-435d-9acb-75f17e0334fb
            client request ID: 9e0fce66-a04c-49ec-96a9-4c1b9377e626
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 15:10	PR #7688	2114736549408018432

alert [svc] KubePodCrashLooping fired

full failure pattern:

alert [svc] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (3)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:55:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.51:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-n8rf5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d66701f9-547c-4900-999a-9cc3801125e4"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-n8rf5 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:55:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.51:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-n8rf5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d66701f9-547c-4900-999a-9cc3801125e4"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-n8rf5 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:55:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r6g8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d6ff7b03-cd37-4dcb-b713-5184927d807f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r6g8v (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r6g8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d6ff7b03-cd37-4dcb-b713-5184927d807f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r6g8v (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:39:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-x7p5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6a5476df-33c8-40f9-ae9f-b4673c8f9f41"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-x7p5s (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:41:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.238:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r248f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="df5123e3-25f8-4237-b3ae-f6ee9c3dc14a"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r248f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:41:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.238:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r248f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="df5123e3-25f8-4237-b3ae-f6ee9c3dc14a"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r248f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:41:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-x7p5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6a5476df-33c8-40f9-ae9f-b4673c8f9f41"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-x7p5s (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:00:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdbj8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cf5b240e-dae8-4a44-acae-3a125b10e344"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdbj8 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:05:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-b6k74", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="259cbe08-9376-4204-b2ee-52499ae8c38f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-b6k74 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:05:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-b6k74", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="259cbe08-9376-4204-b2ee-52499ae8c38f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-b6k74 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:05:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdbj8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cf5b240e-dae8-4a44-acae-3a125b10e344"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdbj8 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-26 14:55	PR #7662	2114732780888788992
2026-10-26 09:27	PR #7712	2114650067662016512
2026-10-26 07:00	PR #7662	2114613195741073408
2026-10-26 05:30	PR #7712	2114590452421234688

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d17e587640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zlz47zdxrs5r" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "name": "7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "status": "Failed",
      "startTime": "2026-10-26T07:28:38.842864831Z",
      "endTime": "2026-10-26T07:47:44.945885677Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
            correlation ID: 08c3cc6c-2bcb-4d71-83fb-d91ac0512763
            client request ID: dc9b7dea-fdfa-46b9-bb12-bc31e259fdc0
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d17e587640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zlz47zdxrs5r" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "name": "7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "status": "Failed",
      "startTime": "2026-10-26T07:28:38.842864831Z",
      "endTime": "2026-10-26T07:47:44.945885677Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
            correlation ID: 08c3cc6c-2bcb-4d71-83fb-d91ac0512763
            client request ID: dc9b7dea-fdfa-46b9-bb12-bc31e259fdc0

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2ca8d1751180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5r2ss57dw6dn" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "name": "7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "status": "Failed",
      "startTime": "2026-10-26T07:10:55.24771548Z",
      "endTime": "2026-10-26T07:29:55.206194815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xtq4l-l6wv8: Failed: availabilitysets failed to create or update. err: failed to get existing resource ingress-cert-le-5r2ss57dw6dn--managed/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5r2ss57dw6dn--managed/providers/Microsoft.Compute/availabilitySets/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
            correlation ID: c6deec64-4edf-4f99-842e-fae9ee3bc0d5
            client request ID: 7c4b8c2b-3bc2-4ab4-98b5-cee20bd9e4da
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2ca8d1751180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5r2ss57dw6dn" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "name": "7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "status": "Failed",
      "startTime": "2026-10-26T07:10:55.24771548Z",
      "endTime": "2026-10-26T07:29:55.206194815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xtq4l-l6wv8: Failed: availabilitysets failed to create or update. err: failed to get existing resource ingress-cert-le-5r2ss57dw6dn--managed/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5r2ss57dw6dn--managed/providers/Microsoft.Compute/availabilitySets/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
            correlation ID: c6deec64-4edf-4f99-842e-fae9ee3bc0d5
            client request ID: 7c4b8c2b-3bc2-4ab4-98b5-cee20bd9e4da

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x3405dea41d00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-mjjkc7tsg6pw" to finish creating: GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "name": "152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "status": "Failed",
      "startTime": "2026-10-26T06:30:24.860566935Z",
      "endTime": "2026-10-26T06:49:25.947090385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
            correlation ID: 4ec64207-73fc-4c1b-8973-f8f47f8d6a7e
            client request ID: e0882a3c-3a9e-4bc0-8c16-c4da5403ce37
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x3405dea41d00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-mjjkc7tsg6pw" to finish creating: GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "name": "152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "status": "Failed",
      "startTime": "2026-10-26T06:30:24.860566935Z",
      "endTime": "2026-10-26T06:49:25.947090385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
            correlation ID: 4ec64207-73fc-4c1b-8973-f8f47f8d6a7e
            client request ID: e0882a3c-3a9e-4bc0-8c16-c4da5403ce37

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-26 06:10	PR #7573	2114600535943286784
2026-10-26 05:30	PR #7712	2114590452421234688

ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (2)

time=2026-10-26T23:19:14.439Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T23:19:15.908Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T23:19:17.082Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=facd493a02be1ad67e57b07e2fbd173b77c816ca0e9fe44bbdba2fb459fbeb85 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5967872-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ffacd493a02be1ad67e57b07e2fbd173b77c816ca0e9fe44bbdba2fb459fbeb85
time=2026-10-26T23:21:47.533Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.Resources/deployments/facd493a02be1ad67e57b07e2fbd173b77c816ca0e9fe44bbdba2fb459fbeb85/operationStatuses/08584085501288714898\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.Resources/deployments/cluster-wsenpawidiq6k\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j5967872-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j5967872-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-26T23:23:47.535Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T23:23:48.327Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T23:23:49.402Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=40b45e9bebcc2cf533d290687c03a069a8f711a71e9ba5f2836d47dbdaf9f362 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5967872-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F40b45e9bebcc2cf533d290687c03a069a8f711a71e9ba5f2836d47dbdaf9f362
time=2026-10-26T23:26:19.774Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.Resources/deployments/40b45e9bebcc2cf533d290687c03a069a8f711a71e9ba5f2836d47dbdaf9f362/operationStatuses/08584085498565391842\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-26T12:51:59.398Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T12:52:00.891Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T12:52:02.109Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=255e1b4b8b6e4372d437a9506a320bc949b0cc72bb4ea9195aae5ed8cedae0ec portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6600576-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F255e1b4b8b6e4372d437a9506a320bc949b0cc72bb4ea9195aae5ed8cedae0ec
time=2026-10-26T12:55:02.788Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.Resources/deployments/255e1b4b8b6e4372d437a9506a320bc949b0cc72bb4ea9195aae5ed8cedae0ec/operationStatuses/08584085877640227854\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.Resources/deployments/cluster-pzvsdmxdkwyhw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j6600576-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j6600576-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-26T12:57:02.789Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T12:57:03.570Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T12:57:05.013Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4a6c02a4a9cb64920875392e566fb8079c7d7a854d89e39f64dab8204f190275 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6600576-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4a6c02a4a9cb64920875392e566fb8079c7d7a854d89e39f64dab8204f190275
time=2026-10-26T12:58:35.328Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.Resources/deployments/4a6c02a4a9cb64920875392e566fb8079c7d7a854d89e39f64dab8204f190275/operationStatuses/08584085874612065441\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-26 22:51	PR #7662	2114852418565967872
2026-10-26 12:31	PR #7754	2114696412376600576

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, kube-scheduler]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x17d7d76a6b60>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n574rs4kgzmb" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "name": "0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:22.647294159Z",
      "endTime": "2026-10-26T04:05:25.39660378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, kube-scheduler, azure-cloud-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
            correlation ID: 2461ce1f-a2e3-4c70-ae70-58c7bc6e13e3
            client request ID: 87ad1faa-4a8b-48cb-888d-882999d849fb
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x17d7d76a6b60>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n574rs4kgzmb" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "name": "0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:22.647294159Z",
      "endTime": "2026-10-26T04:05:25.39660378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, kube-scheduler, azure-cloud-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
            correlation ID: 2461ce1f-a2e3-4c70-ae70-58c7bc6e13e3
            client request ID: 87ad1faa-4a8b-48cb-888d-882999d849fb
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 02:50	PR #7721	2114550200386195456

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x1cab81a88220>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-kfvwp2t79pvw" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403",
      "name": "b86345e2-8212-42f1-a199-1a2d8b540403",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:43.851254184Z",
      "endTime": "2026-10-26T04:05:45.382791214Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
            correlation ID: 63ad8909-3abd-481b-ad9d-99ee8f5ef24a
            client request ID: 2dbc76d4-ed2f-44e9-8bf3-81fac3665977
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x1cab81a88220>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-kfvwp2t79pvw" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403",
      "name": "b86345e2-8212-42f1-a199-1a2d8b540403",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:43.851254184Z",
      "endTime": "2026-10-26T04:05:45.382791214Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
            correlation ID: 63ad8909-3abd-481b-ad9d-99ee8f5ef24a
            client request ID: 2dbc76d4-ed2f-44e9-8bf3-81fac3665977
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 02:50	PR #7721	2114550200386195456

ERROR CODE: DeploymentFailed; detail code ConflictError; detail message A conflict occurred that prevented the operation from completing. The operation failed because the Key Vaul...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ConflictError; detail message A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:78]: failed to create customer resources for ARM64 nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0x240d16aa66e0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-z9s25h" in resourcegroup="arm64-vm-cluster-r2854rq2nbt4" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ConflictError\",\r\n    \"message\": \"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
            correlation ID: 23c397c2-e55d-434d-a806-434e6889bce6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:78]: failed to create customer resources for ARM64 nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0x240d16aa66e0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-z9s25h" in resourcegroup="arm64-vm-cluster-r2854rq2nbt4" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ConflictError\",\r\n    \"message\": \"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
            correlation ID: 23c397c2-e55d-434d-a806-434e6889bce6
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 08:29	PR #7573	2114635485509849088

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-<id>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x37dc0b3be620>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        err: <*errors.errorString | 0x37dc0b40dbb0>{
            s: "dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x37dc0b3be620>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        err: <*errors.errorString | 0x37dc0b40dbb0>{
            s: "dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 22:17	PR #7615	2114843878455513088

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '1' sec... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d"
Unexpected error:
    <*fmt.wrapError | 0x1a5d7d0bada0>: 
    failed to create NodePool np-csnz6d: failed waiting for nodepool="np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d" in resourcegroup="rg-candidate-5-0-x5kw7d69w7fz" to finish creating: GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6",
      "name": "27744955-3d23-4917-a3e3-fea7a42353f6",
      "status": "Failed",
      "startTime": "2026-10-26T22:49:05.815309572Z",
      "endTime": "2026-10-26T23:08:05.399599048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Compute/virtualMachines/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Network/networkInterfaces/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
            correlation ID: aca41f08-bc3a-4eb0-aeee-dbec04407cb6
            client request ID: 503f9ad7-c50a-4e8b-9147-306bbfbcb5a4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d"
Unexpected error:
    <*fmt.wrapError | 0x1a5d7d0bada0>: 
    failed to create NodePool np-csnz6d: failed waiting for nodepool="np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d" in resourcegroup="rg-candidate-5-0-x5kw7d69w7fz" to finish creating: GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6",
      "name": "27744955-3d23-4917-a3e3-fea7a42353f6",
      "status": "Failed",
      "startTime": "2026-10-26T22:49:05.815309572Z",
      "endTime": "2026-10-26T23:08:05.399599048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Compute/virtualMachines/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Network/networkInterfaces/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
            correlation ID: aca41f08-bc3a-4eb0-aeee-dbec04407cb6
            client request ID: 503f9ad7-c50a-4e8b-9147-306bbfbcb5a4
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 21:41	PR #6661	2114834707903418368

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xa75b49b000>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-c7rz5j8lqwzr" to finish creating: GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "name": "116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "status": "Failed",
      "startTime": "2026-10-26T16:04:30.855869275Z",
      "endTime": "2026-10-26T16:23:34.187561349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
            correlation ID: 0413d2ef-0809-4a88-a400-52222573fc14
            client request ID: 2d020f95-fbad-4853-8497-2a3822d5b948
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xa75b49b000>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-c7rz5j8lqwzr" to finish creating: GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "name": "116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "status": "Failed",
      "startTime": "2026-10-26T16:04:30.855869275Z",
      "endTime": "2026-10-26T16:23:34.187561349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
            correlation ID: 0413d2ef-0809-4a88-a400-52222573fc14
            client request ID: 2d020f95-fbad-4853-8497-2a3822d5b948
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 14:55	PR #7662	2114732780888788992

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...

full failure pattern:

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-23-99qq4z"
Unexpected error:
    <*errors.joinError | 0x39eb4f22870>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd": nodes "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x39eb302e180>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x39eb302e120>{
                    msg: "failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x39eb302e100>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x39eb3896140>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-23-99qq4z"
Unexpected error:
    <*errors.joinError | 0x39eb4f22870>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd": nodes "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x39eb302e180>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x39eb302e120>{
                    msg: "failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x39eb302e100>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x39eb3896140>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 17:49	PR #7615	2114776506176638976

not all expected service metrics appeared in Azure Monitor

full failure pattern:

not all expected service metrics appeared in Azure Monitor

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:122]: Timed out after 900.000s.
not all expected service metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:51 with:
expected maestro build info (MGMT) metrics but got no results (query: maestro_build_info)
Expected
    <[]prometheus.Result | len:0, cap:0>: []
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:122]: Timed out after 900.000s.
not all expected service metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:51 with:
expected maestro build info (MGMT) metrics but got no results (query: maestro_build_info)
Expected
    <[]prometheus.Result | len:0, cap:0>: []
not to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from SVC and MGMT clusters to be present in Azure Monitor WOrkspace	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-26 10:29	PR #7665	2114665780556599296