No runs match your search.
Environment: STG
| Time (UTC) | Runtime | Job | Region | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|---|---|
| 1h 33m 56s | periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly post-goodmerged PR |
uksouth | e2e | failed | n/a | 5 | e2e (5)ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-srbb2hbvsczt/cluster-nightly-5-1-vxb9dm should provision
Unexpected error:
<*fmt.wrapError | 0xc000c1af40>:
failed to create HCP cluster cluster-nightly-5-1-vxb9dm: failed waiting for cluster="cluster-nightly-5-1-vxb9dm" in resourcegroup="rg-nightly-5-1-srbb2hbvsczt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
"name": "fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
"status": "Failed",
"startTime": "2026-09-30T23:26:50.189510673Z",
"endTime": "2026-09-30T23:55:57.188151115Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-105558 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-srbb2hbvsczt/cluster-nightly-5-1-vxb9dm should provision
Unexpected error:
<*fmt.wrapError | 0xc000c1af40>:
failed to create HCP cluster cluster-nightly-5-1-vxb9dm: failed waiting for cluster="cluster-nightly-5-1-vxb9dm" in resourcegroup="rg-nightly-5-1-srbb2hbvsczt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
"name": "fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
"status": "Failed",
"startTime": "2026-09-30T23:26:50.189510673Z",
"endTime": "2026-09-30T23:55:57.188151115Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-105558 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-prg87r" for cluster "cluster-nightly-4-20-prg87r"
Unexpected error:
<*fmt.wrapError | 0xc0014907c0>:
failed to create NodePool np-prg87r: failed starting nodepool creation "np-prg87r" for cluster "cluster-nightly-4-20-prg87r" in resourcegroup="rg-nightly-4-20-kjrkpmc2vcnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kjrkpmc2vcnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-prg87r/nodePools/np-prg87r
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-prg87r" for cluster "cluster-nightly-4-20-prg87r"
Unexpected error:
<*fmt.wrapError | 0xc0014907c0>:
failed to create NodePool np-prg87r: failed starting nodepool creation "np-prg87r" for cluster "cluster-nightly-4-20-prg87r" in resourcegroup="rg-nightly-4-20-kjrkpmc2vcnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kjrkpmc2vcnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-prg87r/nodePools/np-prg87r
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
<*fmt.wrapError | 0xc00154c140>:
failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-49bq5x" resourcegroup="rg-np-dg-minor-49bq5x-7f6vt9z24c7x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-49bq5x-7f6vt9z24c7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-49bq5x/nodePools/npdg-422-420
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
<*fmt.wrapError | 0xc00154c140>:
failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-49bq5x" resourcegroup="rg-np-dg-minor-49bq5x-7f6vt9z24c7x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-49bq5x-7f6vt9z24c7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-49bq5x/nodePools/npdg-422-420
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
<*fmt.wrapError | 0xc000b92100>:
failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-qglcmw" in resourcegroup="rg-np-version-upgrade-qglcmw-2xxcc62c56zb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-qglcmw-2xxcc62c56zb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-qglcmw/nodePools/npupgrade-4-20
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
<*fmt.wrapError | 0xc000b92100>:
failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-qglcmw" in resourcegroup="rg-np-version-upgrade-qglcmw-2xxcc62c56zb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-qglcmw-2xxcc62c56zb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-qglcmw/nodePools/npupgrade-4-20
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
<*fmt.wrapError | 0xc000d74240>:
failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-qlzv6l" in resourcegroup="rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qlzv6l/nodePools/nps-420-422
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
<*fmt.wrapError | 0xc000d74240>:
failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-qlzv6l" in resourcegroup="rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qlzv6l/nodePools/nps-420-422
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
...
occurred |
|
| 2h 17m 16s | branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR |
uksouth | e2e | failed | n/a | 5 | e2e (5)ERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-djrg27" for cluster "cluster-candidate-4-20-djrg27"
Unexpected error:
<*fmt.wrapError | 0xc000119c40>:
failed to create NodePool np-djrg27: failed waiting for nodepool="np-djrg27" for cluster "cluster-candidate-4-20-djrg27" in resourcegroup="rg-candidate-4-20-hwlm5v4jsr9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44",
"name": "94e00bc9-969c-49a7-a690-ac544d6d6b44",
"status": "Failed",
"startTime": "2026-09-30T16:46:47.162303776Z",
"endTime": "2026-09-30T17:05:53.284012473Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-djrg27" for cluster "cluster-candidate-4-20-djrg27"
Unexpected error:
<*fmt.wrapError | 0xc000119c40>:
failed to create NodePool np-djrg27: failed waiting for nodepool="np-djrg27" for cluster "cluster-candidate-4-20-djrg27" in resourcegroup="rg-candidate-4-20-hwlm5v4jsr9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44",
"name": "94e00bc9-969c-49a7-a690-ac544d6d6b44",
"status": "Failed",
"startTime": "2026-09-30T16:46:47.162303776Z",
"endTime": "2026-09-30T17:05:53.284012473Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
<*fmt.wrapError | 0xc001703c60>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jmc9c7" in resourcegroup="rg-np-skip-minor-jmc9c7-9dtxqp29vbsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
"name": "d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
"status": "Failed",
"startTime": "2026-09-30T17:21:41.15243756Z",
"endTime": "2026-09-30T17:40:44.044506032Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
<*fmt.wrapError | 0xc001703c60>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jmc9c7" in resourcegroup="rg-np-skip-minor-jmc9c7-9dtxqp29vbsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
"name": "d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
"status": "Failed",
"startTime": "2026-09-30T17:21:41.15243756Z",
"endTime": "2026-09-30T17:40:44.044506032Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredfailed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0xc001116780>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-vh7hggp8cw8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba",
"name": "eed44f57-1a91-4587-8901-935588d355ba",
"status": "Failed",
"startTime": "2026-09-30T17:09:44.769920829Z",
"endTime": "2026-09-30T17:28:44.009160044Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n"
}
}
--------------------------------------------------------------------------------
...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0xc001116780>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-vh7hggp8cw8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba",
"name": "eed44f57-1a91-4587-8901-935588d355ba",
"status": "Failed",
"startTime": "2026-09-30T17:09:44.769920829Z",
"endTime": "2026-09-30T17:28:44.009160044Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n"
}
}
--------------------------------------------------------------------------------
...ERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0xc0015c0ae0>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-twtdf7gdskgb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512",
"name": "ffbf36ab-0bd9-4c0c-8698-3035c7937512",
"status": "Failed",
"startTime": "2026-09-30T16:25:28.987424124Z",
"endTime": "2026-09-30T16:44:30.182558055Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-njlb6-5n7mv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-njlb6-5n7mv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0xc0015c0ae0>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-twtdf7gdskgb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512",
"name": "ffbf36ab-0bd9-4c0c-8698-3035c7937512",
"status": "Failed",
"startTime": "2026-09-30T16:25:28.987424124Z",
"endTime": "2026-09-30T16:44:30.182558055Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-njlb6-5n7mv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-njlb6-5n7mv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
<*fmt.wrapError | 0xc0012c2500>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-xhfvnf" in resourcegroup="rg-np-noedge-xhfvnf-d6cdpdqmxzz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
"name": "79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
"status": "Failed",
"startTime": "2026-09-30T17:20:42.49451817Z",
"endTime": "2026-09-30T17:39:43.993225307Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
<*fmt.wrapError | 0xc0012c2500>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-xhfvnf" in resourcegroup="rg-np-noedge-xhfvnf-d6cdpdqmxzz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
"name": "79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
"status": "Failed",
"startTime": "2026-09-30T17:20:42.49451817Z",
"endTime": "2026-09-30T17:39:43.993225307Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred |
|
| 2h 17m 36s | branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR |
uksouth | e2e | failed | n/a | 4 | e2e (4)ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
<*errors.joinError | 0xc00148e9c0>:
resource group "rg-zstream-upgrade-5-0-l6m7gzggxh5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-mrptx7" in resourcegroup="rg-zstream-upgrade-5-0-l6m7gzggxh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5",
"name": "d96cdefb-0420-40da-ba9f-dde8391c74b5",
"status": "Failed",
"startTime": "2026-09-30T13:28:47.89996524Z",
"endTime": "2026-09-30T13:52:47.080528249Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5bj2pp5rm42moqm553gsgpuq5afnl4 still exists (deletion dispatched at 2026-09-30T13:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
<*errors.joinError | 0xc00148e9c0>:
resource group "rg-zstream-upgrade-5-0-l6m7gzggxh5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-mrptx7" in resourcegroup="rg-zstream-upgrade-5-0-l6m7gzggxh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5",
"name": "d96cdefb-0420-40da-ba9f-dde8391c74b5",
"status": "Failed",
"startTime": "2026-09-30T13:28:47.89996524Z",
"endTime": "2026-09-30T13:52:47.080528249Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5bj2pp5rm42moqm553gsgpuq5afnl4 still exists (deletion dispatched at 2026-09-30T13:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
<*fmt.wrapError | 0xc0009ed920>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-rkbcqh" in resourcegroup="rg-np-skip-minor-rkbcqh-h7clvkbvfrzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
"name": "0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
"status": "Failed",
"startTime": "2026-09-30T13:38:48.671242236Z",
"endTime": "2026-09-30T13:57:57.057793Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
<*fmt.wrapError | 0xc0009ed920>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-rkbcqh" in resourcegroup="rg-np-skip-minor-rkbcqh-h7clvkbvfrzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
"name": "0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
"status": "Failed",
"startTime": "2026-09-30T13:38:48.671242236Z",
"endTime": "2026-09-30T13:57:57.057793Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapError | 0xc000c62580>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4ldsbj98fhm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693",
"name": "d5b038c9-d01f-4cce-8da4-821ff1fad693",
"status": "Failed",
"startTime": "2026-09-30T12:41:58.878961581Z",
"endTime": "2026-09-30T13:01:07.023295945Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapError | 0xc000c62580>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4ldsbj98fhm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693",
"name": "d5b038c9-d01f-4cce-8da4-821ff1fad693",
"status": "Failed",
"startTime": "2026-09-30T12:41:58.878961581Z",
"endTime": "2026-09-30T13:01:07.023295945Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredfailed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
<*fmt.wrapError | 0xc000cd79a0>:
failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ldfdqgv9q9rv" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
<*fmt.wrapError | 0xc000cd79a0>:
failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ldfdqgv9q9rv" to finish creating: context deadline exceeded
...
occurred |
|
| 2h 14m 46s | branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR |
uksouth | e2e | failed | n/a | 2 | e2e (2)ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
<*errors.joinError | 0xc000a8a4b0>:
resource group "rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-b6jrz8" in resourcegroup="rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
"name": "8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
"status": "Failed",
"startTime": "2026-09-30T08:07:19.758629115Z",
"endTime": "2026-09-30T08:31:25.87901072Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56tcneq332c0ht4fv2g22jot5kdluc still exists (deletion dispatched at 2026-09-30T08:07:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
<*errors.joinError | 0xc000a8a4b0>:
resource group "rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-b6jrz8" in resourcegroup="rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
"name": "8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
"status": "Failed",
"startTime": "2026-09-30T08:07:19.758629115Z",
"endTime": "2026-09-30T08:31:25.87901072Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56tcneq332c0ht4fv2g22jot5kdluc still exists (deletion dispatched at 2026-09-30T08:07:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredfailed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0xc0013950a0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6stx4mwz4j4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35",
"name": "626e713d-65ca-4af7-8b7c-fccfc5b04f35",
"status": "Failed",
"startTime": "2026-09-30T08:05:30.277326796Z",
"endTime": "2026-09-30T08:24:30.136180053Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n"
}
}
--------------------------------------------------------------------------------
...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0xc0013950a0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6stx4mwz4j4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35",
"name": "626e713d-65ca-4af7-8b7c-fccfc5b04f35",
"status": "Failed",
"startTime": "2026-09-30T08:05:30.277326796Z",
"endTime": "2026-09-30T08:24:30.136180053Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n"
}
}
--------------------------------------------------------------------------------
... |
|
| 1h 58m 4s | periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR |
uksouth | e2e | failed | n/a | 5 | e2e (5)ERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lssckk" for cluster "cluster-candidate-4-22-lssckk"
Unexpected error:
<*fmt.wrapError | 0xc000d0d380>:
failed to create NodePool np-lssckk: failed waiting for nodepool="np-lssckk" for cluster "cluster-candidate-4-22-lssckk" in resourcegroup="rg-candidate-4-22-bqwkx6ssnq7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8",
"name": "b6149906-5b1c-417e-9e9c-3b2c8a352da8",
"status": "Failed",
"startTime": "2026-09-30T02:38:56.070433953Z",
"endTime": "2026-09-30T02:57:58.4595049Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lssckk" for cluster "cluster-candidate-4-22-lssckk"
Unexpected error:
<*fmt.wrapError | 0xc000d0d380>:
failed to create NodePool np-lssckk: failed waiting for nodepool="np-lssckk" for cluster "cluster-candidate-4-22-lssckk" in resourcegroup="rg-candidate-4-22-bqwkx6ssnq7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8",
"name": "b6149906-5b1c-417e-9e9c-3b2c8a352da8",
"status": "Failed",
"startTime": "2026-09-30T02:38:56.070433953Z",
"endTime": "2026-09-30T02:57:58.4595049Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0xc001035640>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-4kv8jbqzgm6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27",
"name": "0011df08-d47e-405c-b953-36672f501c27",
"status": "Failed",
"startTime": "2026-09-30T03:17:50.983140083Z",
"endTime": "2026-09-30T03:36:56.457139595Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0xc001035640>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-4kv8jbqzgm6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27",
"name": "0011df08-d47e-405c-b953-36672f501c27",
"status": "Failed",
"startTime": "2026-09-30T03:17:50.983140083Z",
"endTime": "2026-09-30T03:36:56.457139595Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0xc0008827a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tb59h9fbmsnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985",
"name": "69cf1894-8af7-4f4d-985a-7d39b5ffb985",
"status": "Failed",
"startTime": "2026-09-30T02:15:06.409498278Z",
"endTime": "2026-09-30T02:34:10.118150935Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0xc0008827a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tb59h9fbmsnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985",
"name": "69cf1894-8af7-4f4d-985a-7d39b5ffb985",
"status": "Failed",
"startTime": "2026-09-30T02:15:06.409498278Z",
"endTime": "2026-09-30T02:34:10.118150935Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurredVerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
<*errors.joinError | 0xc0016d54b8>:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
<*errors.joinError | 0xc0016d54b8>:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
...ERROR CODE: DeadlineExceeded; provider Microsoft.Compute Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0xc000c25d80>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jtns6kb9n72s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521",
"name": "13d22eca-d7af-43b2-9ab7-e630c81c0521",
"status": "Failed",
"startTime": "2026-09-30T02:16:05.748001285Z",
"endTime": "2026-09-30T02:35:10.119119459Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0xc000c25d80>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jtns6kb9n72s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521",
"name": "13d22eca-d7af-43b2-9ab7-e630c81c0521",
"status": "Failed",
"startTime": "2026-09-30T02:16:05.748001285Z",
"endTime": "2026-09-30T02:35:10.119119459Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n"
}
}
--------------------------------------------------------------------------------
...
occurred |