No runs match your search.
Environment: PROD
| Time (UTC) | Job | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 3 | e2e (3)tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: Unexpected error:
<*errors.joinError | 0xc0019264e0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
VerifyBasicAccess failed: failed to list services: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: Unexpected error:
<*errors.joinError | 0xc0019264e0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
VerifyBasicAccess failed: failed to list services: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
...
occurredtls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:127]: Unexpected error:
<*errors.joinError | 0xc0007cd3c8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
VerifyBasicAccess failed: failed to list services: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:127]: Unexpected error:
<*errors.joinError | 0xc0007cd3c8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
VerifyBasicAccess failed: failed to list services: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
...
occurredclient rate limiter Wait returned an error: context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc000e508b8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fjrggl" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc000e508b8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fjrggl" to finish creating: context deadline exceeded
...
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 3 | e2e (3)ERROR CODE: ConflictingConcurrentWriteNotAllowed; detail message The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later.; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:249]: Unexpected error:
<*errors.joinError | 0xc00171e978>:
failed to cleanup resource group: ordered cleanup workflow failed for disabled-image-registry-l5b2fs: Delete non-networking resources: failed deleting disabled-image-registry-hcp-cluster (Microsoft.RedHatOpenShift/HCPOpenShiftClusters): DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-l5b2fs/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/disabled-image-registry-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ConflictingConcurrentWriteNotAllowed
--------------------------------------------------------------------------------
{
"error": {
"code": "ConflictingConcurrentWriteNotAllowed",
"message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:249]: Unexpected error:
<*errors.joinError | 0xc00171e978>:
failed to cleanup resource group: ordered cleanup workflow failed for disabled-image-registry-l5b2fs: Delete non-networking resources: failed deleting disabled-image-registry-hcp-cluster (Microsoft.RedHatOpenShift/HCPOpenShiftClusters): DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-l5b2fs/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/disabled-image-registry-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ConflictingConcurrentWriteNotAllowed
--------------------------------------------------------------------------------
{
"error": {
"code": "ConflictingConcurrentWriteNotAllowed",
"message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
}
}
--------------------------------------------------------------------------------
...
occurredclient rate limiter Wait returned an error: context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc00104c6a8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9vtz75" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc00104c6a8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9vtz75" to finish creating: context deadline exceeded
...
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
<*fmt.wrapErrors | 0xc000910630>:
failed to create HCP cluster cp-ystream-upgrade-4-21-kwwz97, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: failed waiting for cluster="cp-ystream-upgrade-4-21-kwwz97" in resourcegroup="rg-cp-ystream-upgrade-4-21-t2vg4j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
<*fmt.wrapErrors | 0xc000910630>:
failed to create HCP cluster cp-ystream-upgrade-4-21-kwwz97, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: failed waiting for cluster="cp-ystream-upgrade-4-21-kwwz97" in resourcegroup="rg-cp-ystream-upgrade-4-21-t2vg4j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: context deadline exceeded
...
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 2 | e2e (2)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
<*fmt.wrapErrors | 0xc0010f82d0>:
failed to create HCP cluster np-version-upgrade-cluster-qf6tsk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: failed waiting for cluster="np-version-upgrade-cluster-qf6tsk" in resourcegroup="rg-np-version-upgrade-qf6tsk-8dlq72" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
<*fmt.wrapErrors | 0xc0010f82d0>:
failed to create HCP cluster np-version-upgrade-cluster-qf6tsk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: failed waiting for cluster="np-version-upgrade-cluster-qf6tsk" in resourcegroup="rg-np-version-upgrade-qf6tsk-8dlq72" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: context deadline exceeded
...
occurredclient rate limiter Wait returned an error: context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc00196a408>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hk2d79" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc00196a408>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hk2d79" to finish creating: context deadline exceeded
...
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 3 | e2e (3)ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
<*fmt.wrapError | 0xc0001bfaa0>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xv5wb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361",
"name": "35394bfc-4a06-43b3-aa53-a1d60828c361",
"status": "Failed",
"startTime": "2026-04-22T09:47:15.284582065Z",
"endTime": "2026-04-22T09:48:23.845505796Z",
"error": {
"code": "InternalServerError"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
<*fmt.wrapError | 0xc0001bfaa0>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xv5wb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361",
"name": "35394bfc-4a06-43b3-aa53-a1d60828c361",
"status": "Failed",
"startTime": "2026-04-22T09:47:15.284582065Z",
"endTime": "2026-04-22T09:48:23.845505796Z",
"error": {
"code": "InternalServerError"
}
}
--------------------------------------------------------------------------------
...
occurredtls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:141]: Unexpected error:
<*errors.joinError | 0xc000a46d38>:
VerifyBasicAccess failed: failed to list services: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:141]: Unexpected error:
<*errors.joinError | 0xc000a46d38>:
VerifyBasicAccess failed: failed to list services: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
...
occurredclient rate limiter Wait returned an error: context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc000e6a360>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bkrk7k" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc000e6a360>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bkrk7k" to finish creating: context deadline exceeded
...
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a |