<
Single Day: Apr 22
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 3
e2e (3)
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
18:48:36 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: Unexpected error:
    <*errors.joinError | 0xc0019264e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: Unexpected error:
    <*errors.joinError | 0xc0019264e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.z8b5g9a8l3e9r0o.idqn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.z8b5g9a8l3e9r0o.bcp0.uksouth.aroapp-hcp.io
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
18:48:36 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:127]: Unexpected error:
    <*errors.joinError | 0xc0007cd3c8>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:127]: Unexpected error:
    <*errors.joinError | 0xc0007cd3c8>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.i1s3o9k1e1z3u0n.4xyn.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.i1s3o9k1e1z3u0n.4nkk.uksouth.aroapp-hcp.io
    ...
occurred
client rate limiter Wait returned an error: context deadline exceeded
18:48:36 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc000e508b8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fjrggl" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc000e508b8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fjrggl" to finish creating: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 3
e2e (3)
ERROR CODE: ConflictingConcurrentWriteNotAllowed; detail message The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later.; provider Microsoft.RedHatOpenShift
14:16:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:249]: Unexpected error:
    <*errors.joinError | 0xc00171e978>: 
    failed to cleanup resource group: ordered cleanup workflow failed for disabled-image-registry-l5b2fs: Delete non-networking resources: failed deleting disabled-image-registry-hcp-cluster (Microsoft.RedHatOpenShift/HCPOpenShiftClusters): DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-l5b2fs/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: ConflictingConcurrentWriteNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ConflictingConcurrentWriteNotAllowed",
        "message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:249]: Unexpected error:
    <*errors.joinError | 0xc00171e978>: 
    failed to cleanup resource group: ordered cleanup workflow failed for disabled-image-registry-l5b2fs: Delete non-networking resources: failed deleting disabled-image-registry-hcp-cluster (Microsoft.RedHatOpenShift/HCPOpenShiftClusters): DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-l5b2fs/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: ConflictingConcurrentWriteNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ConflictingConcurrentWriteNotAllowed",
        "message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
client rate limiter Wait returned an error: context deadline exceeded
14:16:47 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc00104c6a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9vtz75" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc00104c6a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9vtz75" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
14:16:47 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000910630>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-kwwz97, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: failed waiting for cluster="cp-ystream-upgrade-4-21-kwwz97" in resourcegroup="rg-cp-ystream-upgrade-4-21-t2vg4j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000910630>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-kwwz97, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: failed waiting for cluster="cp-ystream-upgrade-4-21-kwwz97" in resourcegroup="rg-cp-ystream-upgrade-4-21-t2vg4j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-kwwz97 in resource group rg-cp-ystream-upgrade-4-21-t2vg4j, error: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:28:31 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010f82d0>: 
    failed to create HCP cluster np-version-upgrade-cluster-qf6tsk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: failed waiting for cluster="np-version-upgrade-cluster-qf6tsk" in resourcegroup="rg-np-version-upgrade-qf6tsk-8dlq72" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010f82d0>: 
    failed to create HCP cluster np-version-upgrade-cluster-qf6tsk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: failed waiting for cluster="np-version-upgrade-cluster-qf6tsk" in resourcegroup="rg-np-version-upgrade-qf6tsk-8dlq72" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qf6tsk in resource group rg-np-version-upgrade-qf6tsk-8dlq72, error: context deadline exceeded
    ...
occurred
client rate limiter Wait returned an error: context deadline exceeded
11:28:31 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc00196a408>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hk2d79" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc00196a408>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hk2d79" to finish creating: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 3
e2e (3)
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
09:10:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc0001bfaa0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xv5wb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361",
      "name": "35394bfc-4a06-43b3-aa53-a1d60828c361",
      "status": "Failed",
      "startTime": "2026-04-22T09:47:15.284582065Z",
      "endTime": "2026-04-22T09:48:23.845505796Z",
      "error": {
        "code": "InternalServerError"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc0001bfaa0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xv5wb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35394bfc-4a06-43b3-aa53-a1d60828c361",
      "name": "35394bfc-4a06-43b3-aa53-a1d60828c361",
      "status": "Failed",
      "startTime": "2026-04-22T09:47:15.284582065Z",
      "endTime": "2026-04-22T09:48:23.845505796Z",
      "error": {
        "code": "InternalServerError"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
09:10:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:141]: Unexpected error:
    <*errors.joinError | 0xc000a46d38>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:141]: Unexpected error:
    <*errors.joinError | 0xc000a46d38>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.s5s7v5v8m6t7f6o.wmw2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.s5s7v5v8m6t7f6o.mp5p.uksouth.aroapp-hcp.io
    ...
occurred
client rate limiter Wait returned an error: context deadline exceeded
09:10:33 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc000e6a360>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bkrk7k" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc000e6a360>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bkrk7k" to finish creating: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a