CIHealth
<
Single Day: Apr 23▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 7
e2e (7)
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
23:08:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:103]: Unexpected error:
    <*errors.joinError | 0xc001984888>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.q5l8i2i0z6e7p8t.lohc.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.q5l8i2i0z6e7p8t.lohc.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:103]: Unexpected error:
    <*errors.joinError | 0xc001984888>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.q5l8i2i0z6e7p8t.lohc.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.q5l8i2i0z6e7p8t.lohc.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q5l8i2i0z6e7p8t.8azc.uksouth.aroapp-hcp.io
    ...
occurred
MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equal
23:08:27 UTC · failed at e2e · test MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
23:08:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:160]: Unexpected error:
    <*errors.joinError | 0xc00104ab10>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:160]: Unexpected error:
    <*errors.joinError | 0xc00104ab10>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
23:08:27 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:161]: Unexpected error:
    <*url.Error | 0xc00174e600>: 
    Get "https://api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.t9i0s9f7m4g0f5e.rzw9.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:161]: Unexpected error:
    <*url.Error | 0xc00174e600>: 
    Get "https://api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.t9i0s9f7m4g0f5e.rzw9.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
23:08:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:127]: Unexpected error:
    <*errors.joinError | 0xc000ef6960>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.a3p3i1v4e6u2c5w.e9wf.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.a3p3i1v4e6u2c5w.e9wf.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:127]: Unexpected error:
    <*errors.joinError | 0xc000ef6960>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.a3p3i1v4e6u2c5w.e9wf.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.a3p3i1v4e6u2c5w.e9wf.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.a3p3i1v4e6u2c5w.v3e1.uksouth.aroapp-hcp.io
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
23:08:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:108]: Unexpected error:
    <*errors.joinError | 0xc000d7c2a0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.l1h0l2r4w5p9y7a.hr9b.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.l1h0l2r4w5p9y7a.hr9b.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:108]: Unexpected error:
    <*errors.joinError | 0xc000d7c2a0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.l1h0l2r4w5p9y7a.hr9b.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.l1h0l2r4w5p9y7a.hr9b.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.l1h0l2r4w5p9y7a.jwmn.uksouth.aroapp-hcp.io
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
23:08:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
    <*errors.joinError | 0xc000f427e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
    VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
    <*errors.joinError | 0xc000f427e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
    VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 5
e2e (5)
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
14:57:24 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Unexpected error:
    <*errors.joinError | 0xc000012cd8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t9y2f8k0a0s5v9s.5zke.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t9y2f8k0a0s5v9s.5zke.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Unexpected error:
    <*errors.joinError | 0xc000012cd8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t9y2f8k0a0s5v9s.5zke.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t9y2f8k0a0s5v9s.5zke.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9y2f8k0a0s5v9s.ksq8.uksouth.aroapp-hcp.io
    ...
occurred
MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equal
14:57:24 UTC · failed at e2e · test MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
14:57:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
    <*errors.joinError | 0xc001026d98>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t3j0r5a0h5f9i5x.wsa2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t3j0r5a0h5f9i5x.wsa2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io
    VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.t3j0r5a0h5f9i5x.wsa2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
    <*errors.joinError | 0xc001026d98>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t3j0r5a0h5f9i5x.wsa2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t3j0r5a0h5f9i5x.wsa2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io
    VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.t3j0r5a0h5f9i5x.wsa2.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t3j0r5a0h5f9i5x.7i35.uksouth.aroapp-hcp.io
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
14:57:24 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0008ec4b0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-htjcmf, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-htjcmf" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0008ec4b0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-htjcmf, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-htjcmf" to finish creating: context deadline exceeded
    ...
occurred
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>
14:57:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*errors.joinError | 0xc001792990>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.x1r8i2z2l1w4b6m.7c6c.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.x1r8i2z2l1w4b6m.7c6c.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*errors.joinError | 0xc001792990>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.x1r8i2z2l1w4b6m.7c6c.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.x1r8i2z2l1w4b6m.7c6c.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.x1r8i2z2l1w4b6m.se7s.uksouth.aroapp-hcp.io
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
client rate limiter Wait returned an error: context deadline exceeded
08:28:16 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc000060d20>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-wt4d9m" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc000060d20>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-wt4d9m" to finish creating: context deadline exceeded
    ...
occurred
MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equal
08:28:16 UTC · failed at e2e · test MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 9
e2e (9)
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:08:03 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e26720>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-wxgvk4, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-wxgvk4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-wxgvk4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e26720>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-wxgvk4, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-wxgvk4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-wxgvk4, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:08:03 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc0017b0570>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fsqjp5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc0017b0570>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fsqjp5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:08:03 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ffc240>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-pl5xgz, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-pl5xgz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-pl5xgz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ffc240>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-pl5xgz, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-pl5xgz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-pl5xgz, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:08:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc001592840>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-rpl9lv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc001592840>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-rpl9lv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: context deadline exceeded
    ...
occurred
context deadline exceeded
00:08:03 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Cluster should become ready within 45 minutes
00:08:03 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: ...
to be true
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:08:03 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010783f0>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-q29rcv, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-q29rcv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-q29rcv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010783f0>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-q29rcv, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-q29rcv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-q29rcv, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:08:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fd6d50>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-qsxr69" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fd6d50>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-qsxr69" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: context deadline exceeded
    ...
occurred
client rate limiter Wait returned an error: context deadline exceeded
00:08:03 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc001242648>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-qkvnrj" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
    <*errors.joinError | 0xc001242648>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-qkvnrj" to finish creating: context deadline exceeded
    ...
occurred