No runs match your search.
Environment: PROD
| Time (UTC) | Job | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a | |
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
unknown | failed | n/a | 1 | non-artifact-backed failure (no junit artifacts) | |
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a | |
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a | |
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 1 | e2e (1)ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0xc00093afe0>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-bm5z6ztxsxcf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d",
"name": "56042e31-0efc-4287-925a-c64efa21388d",
"status": "Failed",
"startTime": "2026-08-25T10:30:08.889821647Z",
"endTime": "2026-08-25T10:49:13.717723647Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0xc00093afe0>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-bm5z6ztxsxcf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d",
"name": "56042e31-0efc-4287-925a-c64efa21388d",
"status": "Failed",
"startTime": "2026-08-25T10:30:08.889821647Z",
"endTime": "2026-08-25T10:49:13.717723647Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
}
}
--------------------------------------------------------------------------------
...
occurred |
|
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 2 | e2e (2)ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0xc0006d1700>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dnqrhz4h5n9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c",
"name": "26ec1bc7-0ddd-42ef-916c-12f4d896455c",
"status": "Failed",
"startTime": "2026-08-25T07:34:58.109914662Z",
"endTime": "2026-08-25T07:53:59.934466955Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0xc0006d1700>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dnqrhz4h5n9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c",
"name": "26ec1bc7-0ddd-42ef-916c-12f4d896455c",
"status": "Failed",
"startTime": "2026-08-25T07:34:58.109914662Z",
"endTime": "2026-08-25T07:53:59.934466955Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xc000a308e0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ks6pxxnlhsjm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f",
"name": "fc3b6654-559a-489d-a192-908217a06b8f",
"status": "Failed",
"startTime": "2026-08-25T07:33:44.141359446Z",
"endTime": "2026-08-25T07:52:49.875150436Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xc000a308e0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ks6pxxnlhsjm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f",
"name": "fc3b6654-559a-489d-a192-908217a06b8f",
"status": "Failed",
"startTime": "2026-08-25T07:33:44.141359446Z",
"endTime": "2026-08-25T07:52:49.875150436Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
}
}
--------------------------------------------------------------------------------
...
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 5 | e2e (5)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zmwf7l76v4hv/cluster-candidate-5-0-p2dtvg should provision
Unexpected error:
<*fmt.wrapErrors | 0xc000a42420>:
failed to create HCP cluster cluster-candidate-5-0-p2dtvg, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: failed waiting for cluster="cluster-candidate-5-0-p2dtvg" in resourcegroup="rg-candidate-5-0-zmwf7l76v4hv" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zmwf7l76v4hv/cluster-candidate-5-0-p2dtvg should provision
Unexpected error:
<*fmt.wrapErrors | 0xc000a42420>:
failed to create HCP cluster cluster-candidate-5-0-p2dtvg, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: failed waiting for cluster="cluster-candidate-5-0-p2dtvg" in resourcegroup="rg-candidate-5-0-zmwf7l76v4hv" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: context deadline exceeded
...
occurredERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.30; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-hc729b" to minor version 4.21
Unexpected error:
<*exported.ResponseError | 0xc000cc7e30>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4j8dsngwmfpg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-hc729b
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be at least 4.21.30",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-hc729b" to minor version 4.21
Unexpected error:
<*exported.ResponseError | 0xc000cc7e30>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4j8dsngwmfpg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-hc729b
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be at least 4.21.30",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurredVerifyHostedControlPlaneZStreamUpgradeOnly(initial=5 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.008s.
Expected success, but got an error:
<*errors.joinError | 0xc000b1c2d0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.008s.
Expected success, but got an error:
<*errors.joinError | 0xc000b1c2d0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
...ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.35; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-2m6hjk" to minor version 4.20
Unexpected error:
<*exported.ResponseError | 0xc00144a750>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6dbxgwtzxfsh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2m6hjk
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20\": must be at least 4.20.35",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-2m6hjk" to minor version 4.20
Unexpected error:
<*exported.ResponseError | 0xc00144a750>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6dbxgwtzxfsh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2m6hjk
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.20\": must be at least 4.20.35",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurredERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.10; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-qp64bq" to minor version 4.22
Unexpected error:
<*exported.ResponseError | 0xc000a2bec0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-dqpzmwzqshp6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-qp64bq
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.22\": must be at least 4.22.10",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-qp64bq" to minor version 4.22
Unexpected error:
<*exported.ResponseError | 0xc000a2bec0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-dqpzmwzqshp6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-qp64bq
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.22\": must be at least 4.22.10",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurred |
|
| branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a |