CIHealth Run Log

Environment: STG

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:24:25	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	7	e2e (7) timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error: <fmt.wrapErrors \| 0xc0015155c0>: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wdjndj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-wdjndj, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-wdjndj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-wdjndj, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00076c300>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-wdjndj", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error: <fmt.wrapErrors \| 0xc0015155c0>: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wdjndj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-wdjndj, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-wdjndj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-wdjndj, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00076c300>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-wdjndj", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc0015779e0>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-x5mmq2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-x5mmq2, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-x5mmq2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-x5mmq2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a88100>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-x5mmq2", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc0015779e0>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-x5mmq2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-x5mmq2, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-x5mmq2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-x5mmq2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a88100>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-x5mmq2", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc00156a3c0>: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-x5nbs9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-x5nbs9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x5nbs9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-x5nbs9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000dbddd0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-x5nbs9", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc00156a3c0>: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-x5nbs9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-x5nbs9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x5nbs9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-x5nbs9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000dbddd0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-x5nbs9", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapErrors \| 0xc0009decc0>: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-782d8g" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-782d8g, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-782d8g\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-782d8g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000c04df0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-782d8g", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapErrors \| 0xc0009decc0>: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-782d8g" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-782d8g, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-782d8g\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-782d8g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000c04df0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-782d8g", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc0011e3890>: failed waiting for hcpCluster="cluster-stable-4-20-vtmlht" in resourcegroup="rg-stable-4-20-jdtn2v" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-vtmlht in resource group rg-stable-4-20-jdtn2v, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-vtmlht\" in resourcegroup=\"rg-stable-4-20-jdtn2v\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-vtmlht in resource group rg-stable-4-20-jdtn2v, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013502e0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-vtmlht in resource group rg-stable-4-20-jdtn2v", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc0011e3890>: failed waiting for hcpCluster="cluster-stable-4-20-vtmlht" in resourcegroup="rg-stable-4-20-jdtn2v" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-vtmlht in resource group rg-stable-4-20-jdtn2v, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-vtmlht\" in resourcegroup=\"rg-stable-4-20-jdtn2v\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-vtmlht in resource group rg-stable-4-20-jdtn2v, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013502e0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-vtmlht in resource group rg-stable-4-20-jdtn2v", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:245]: Unexpected error: <fmt.wrapErrors \| 0xc001065020>: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-x8t6s2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-x8t6s2, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-x8t6s2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-x8t6s2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0005f7800>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-x8t6s2", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:245]: Unexpected error: <fmt.wrapErrors \| 0xc001065020>: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-x8t6s2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-x8t6s2, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-x8t6s2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-x8t6s2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0005f7800>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-x8t6s2", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 23:24:25 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error: <fmt.wrapErrors \| 0xc00071ba70>: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dmzpzs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-dmzpzs, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-dmzpzs\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-dmzpzs, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0009c5c60>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-dmzpzs", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error: <fmt.wrapErrors \| 0xc00071ba70>: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dmzpzs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-dmzpzs, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-dmzpzs\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-dmzpzs, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0009c5c60>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-dmzpzs", }, <context.deadlineExceededError>{}, ], } occurred
20:40:02	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	14	e2e (14) timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:245]: Unexpected error: <fmt.wrapErrors \| 0xc000e86750>: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-c7npm9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-c7npm9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-c7npm9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-c7npm9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0003412b0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-c7npm9", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:245]: Unexpected error: <fmt.wrapErrors \| 0xc000e86750>: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-c7npm9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-c7npm9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-c7npm9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-c7npm9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0003412b0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-c7npm9", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc0008caba0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-rcgc8p" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-rcgc8p, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-rcgc8p\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-rcgc8p, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00097eb20>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-rcgc8p", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc0008caba0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-rcgc8p" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-rcgc8p, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-rcgc8p\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-rcgc8p, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00097eb20>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-rcgc8p", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc001784cf0>: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-q97v42" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-q97v42, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-q97v42\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-q97v42, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00095a880>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-q97v42", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc001784cf0>: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-q97v42" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-q97v42, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-q97v42\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-q97v42, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00095a880>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-q97v42", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error: <fmt.wrapErrors \| 0xc0011f2b40>: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-rpmtxr" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rpmtxr, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rpmtxr\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rpmtxr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008cf150>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rpmtxr", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error: <fmt.wrapErrors \| 0xc0011f2b40>: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-rpmtxr" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rpmtxr, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rpmtxr\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rpmtxr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008cf150>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rpmtxr", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc00160aa20>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-sbvxnd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sbvxnd, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-sbvxnd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sbvxnd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0003b9b70>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sbvxnd", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc00160aa20>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-sbvxnd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sbvxnd, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-sbvxnd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sbvxnd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0003b9b70>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sbvxnd", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapErrors \| 0xc00178a840>: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kh6vnx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-kh6vnx, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-kh6vnx\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-kh6vnx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000670d20>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-kh6vnx", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapErrors \| 0xc00178a840>: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kh6vnx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-kh6vnx, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-kh6vnx\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-kh6vnx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000670d20>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-kh6vnx", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc00103eb70>: failed waiting for hcpCluster="cluster-stable-4-20-fbtnrz" in resourcegroup="rg-stable-4-20-bcpmj9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-fbtnrz in resource group rg-stable-4-20-bcpmj9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-fbtnrz\" in resourcegroup=\"rg-stable-4-20-bcpmj9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-fbtnrz in resource group rg-stable-4-20-bcpmj9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a221e0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-fbtnrz in resource group rg-stable-4-20-bcpmj9", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc00103eb70>: failed waiting for hcpCluster="cluster-stable-4-20-fbtnrz" in resourcegroup="rg-stable-4-20-bcpmj9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-fbtnrz in resource group rg-stable-4-20-bcpmj9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-fbtnrz\" in resourcegroup=\"rg-stable-4-20-bcpmj9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-fbtnrz in resource group rg-stable-4-20-bcpmj9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a221e0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-fbtnrz in resource group rg-stable-4-20-bcpmj9", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc001208660>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jxthrg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jxthrg, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jxthrg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jxthrg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001486620>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jxthrg", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc001208660>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jxthrg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jxthrg, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jxthrg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jxthrg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001486620>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jxthrg", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error: <fmt.wrapErrors \| 0xc0009612f0>: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-zccvrs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-zccvrs, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zccvrs\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-zccvrs, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013b52e0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-zccvrs", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error: <fmt.wrapErrors \| 0xc0009612f0>: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-zccvrs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-zccvrs, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zccvrs\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-zccvrs, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013b52e0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-zccvrs", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0012cc420>: failed waiting for hcpCluster="cluster-zstream-4-21-zfhpmw" in resourcegroup="rg-zstream-upgrade-4-21-lzhvk4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-zfhpmw in resource group rg-zstream-upgrade-4-21-lzhvk4, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-zfhpmw\" in resourcegroup=\"rg-zstream-upgrade-4-21-lzhvk4\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-zfhpmw in resource group rg-zstream-upgrade-4-21-lzhvk4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00153ccf0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-zfhpmw in resource group rg-zstream-upgrade-4-21-lzhvk4", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0012cc420>: failed waiting for hcpCluster="cluster-zstream-4-21-zfhpmw" in resourcegroup="rg-zstream-upgrade-4-21-lzhvk4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-zfhpmw in resource group rg-zstream-upgrade-4-21-lzhvk4, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-zfhpmw\" in resourcegroup=\"rg-zstream-upgrade-4-21-lzhvk4\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-zfhpmw in resource group rg-zstream-upgrade-4-21-lzhvk4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00153ccf0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-zfhpmw in resource group rg-zstream-upgrade-4-21-lzhvk4", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc0016a0f60>: failed waiting for hcpCluster="admin-cred-lifecycle-9sv9gw" in resourcegroup="admin-credential-lifecycle-test-g6kp9g" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-9sv9gw in resource group admin-credential-lifecycle-test-g6kp9g, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-9sv9gw\" in resourcegroup=\"admin-credential-lifecycle-test-g6kp9g\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-9sv9gw in resource group admin-credential-lifecycle-test-g6kp9g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000986060>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-9sv9gw in resource group admin-credential-lifecycle-test-g6kp9g", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc0016a0f60>: failed waiting for hcpCluster="admin-cred-lifecycle-9sv9gw" in resourcegroup="admin-credential-lifecycle-test-g6kp9g" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-9sv9gw in resource group admin-credential-lifecycle-test-g6kp9g, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-9sv9gw\" in resourcegroup=\"admin-credential-lifecycle-test-g6kp9g\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-9sv9gw in resource group admin-credential-lifecycle-test-g6kp9g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000986060>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-9sv9gw in resource group admin-credential-lifecycle-test-g6kp9g", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error: <fmt.wrapErrors \| 0xc000a3c3c0>: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cw7dq4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-cw7dq4, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cw7dq4\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-cw7dq4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0009e6fa0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-cw7dq4", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error: <fmt.wrapErrors \| 0xc000a3c3c0>: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cw7dq4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-cw7dq4, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cw7dq4\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-cw7dq4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0009e6fa0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-cw7dq4", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc0015b4e40>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-b5bsvg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-b5bsvg, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-b5bsvg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-b5bsvg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0000d3c50>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-b5bsvg", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc0015b4e40>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-b5bsvg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-b5bsvg, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-b5bsvg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-b5bsvg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0000d3c50>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-b5bsvg", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:40:02 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc00071e810>: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-wmxhsh" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-wmxhsh, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-wmxhsh\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-wmxhsh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000fd5a50>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-wmxhsh", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc00071e810>: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-wmxhsh" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-wmxhsh, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-wmxhsh\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-wmxhsh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000fd5a50>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-wmxhsh", }, <context.deadlineExceededError>{}, ], } occurred
17:14:28	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	5	e2e (5) ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 17:14:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapError \| 0xc0008d7580>: failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster="cluster-zstream-4-19-456qg5" in resourcegroup="rg-zstream-upgrade-4-19-p762zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a", "name": "78ee839b-27e8-45ab-9bef-ab361680e06a", "status": "Failed", "startTime": "2026-04-02T17:27:46.835065402Z", "endTime": "2026-04-02T17:27:57.932415489Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008d7560>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001317380>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Content-Length": ["499"], "X-Ms-Request-Id": [ "f6c45648-41ff-430b-8947-420c54510b28", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T172807Z:e6630a99-063f-431a-9926-65553b1b3548", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/26acb1f6-ebb4-4c9f-9e33-8c891414cd26", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Msedge-Ref": [ "Ref A: C88F82DD8BF54F1BA57C9FED21B63FDF Ref B: CHI261140514034 Ref C: 2026-04-02T17:28:07Z", ], "Date": [ "Thu, 02 Apr 2026 17:28:07 GMT", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapError \| 0xc0008d7580>: failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster="cluster-zstream-4-19-456qg5" in resourcegroup="rg-zstream-upgrade-4-19-p762zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a", "name": "78ee839b-27e8-45ab-9bef-ab361680e06a", "status": "Failed", "startTime": "2026-04-02T17:27:46.835065402Z", "endTime": "2026-04-02T17:27:57.932415489Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008d7560>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001317380>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Content-Length": ["499"], "X-Ms-Request-Id": [ "f6c45648-41ff-430b-8947-420c54510b28", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T172807Z:e6630a99-063f-431a-9926-65553b1b3548", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/26acb1f6-ebb4-4c9f-9e33-8c891414cd26", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Msedge-Ref": [ "Ref A: C88F82DD8BF54F1BA57C9FED21B63FDF Ref B: CHI261140514034 Ref C: 2026-04-02T17:28:07Z", ], "Date": [ "Thu, 02 Apr 2026 17:28:07 GMT", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 17:14:28 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc0004066e0>: failed to create HCP cluster np-version-upgrade-cluster-b8fnr6: failed waiting for cluster="np-version-upgrade-cluster-b8fnr6" in resourcegroup="rg-np-version-upgrade-b8fnr6-jjtft5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014", "name": "32d8b3c7-04b2-4a25-9b60-ad0cd12fd014", "status": "Failed", "startTime": "2026-04-02T17:22:57.156170095Z", "endTime": "2026-04-02T17:23:13.582154512Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-b8fnr6: failed waiting for cluster=\"np-version-upgrade-cluster-b8fnr6\" in resourcegroup=\"rg-np-version-upgrade-b8fnr6-jjtft5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"name\": \"32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:57.156170095Z\",\n \"endTime\": \"2026-04-02T17:23:13.582154512Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004066c0>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-b8fnr6\" in resourcegroup=\"rg-np-version-upgrade-b8fnr6-jjtft5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"name\": \"32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:57.156170095Z\",\n \"endTime\": \"2026-04-02T17:23:13.582154512Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001438780>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "X-Ms-Correlation-Request-Id": [ "a49f00f4-104e-4fc7-8144-4b711f384af9", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 0EF81C0355DE4A3EBDA3CF679685BB01 Ref B: CH1AA2020617023 Ref C: 2026-04-02T17:23:18Z", ], "Expires": ["-1"], "X-Ms-Request-Id": [ "fa648e5b-e803-4818-89e0-4eeb3ec28e06", ], "X-Envoy-Upstream-Service-Time": ["10"], "Pragma": ["no-cache"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc0004066e0>: failed to create HCP cluster np-version-upgrade-cluster-b8fnr6: failed waiting for cluster="np-version-upgrade-cluster-b8fnr6" in resourcegroup="rg-np-version-upgrade-b8fnr6-jjtft5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014", "name": "32d8b3c7-04b2-4a25-9b60-ad0cd12fd014", "status": "Failed", "startTime": "2026-04-02T17:22:57.156170095Z", "endTime": "2026-04-02T17:23:13.582154512Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-b8fnr6: failed waiting for cluster=\"np-version-upgrade-cluster-b8fnr6\" in resourcegroup=\"rg-np-version-upgrade-b8fnr6-jjtft5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"name\": \"32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:57.156170095Z\",\n \"endTime\": \"2026-04-02T17:23:13.582154512Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004066c0>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-b8fnr6\" in resourcegroup=\"rg-np-version-upgrade-b8fnr6-jjtft5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"name\": \"32d8b3c7-04b2-4a25-9b60-ad0cd12fd014\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:57.156170095Z\",\n \"endTime\": \"2026-04-02T17:23:13.582154512Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001438780>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "X-Ms-Correlation-Request-Id": [ "a49f00f4-104e-4fc7-8144-4b711f384af9", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 0EF81C0355DE4A3EBDA3CF679685BB01 Ref B: CH1AA2020617023 Ref C: 2026-04-02T17:23:18Z", ], "Expires": ["-1"], "X-Ms-Request-Id": [ "fa648e5b-e803-4818-89e0-4eeb3ec28e06", ], "X-Envoy-Upstream-Service-Time": ["10"], "Pragma": ["no-cache"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 17:14:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapError \| 0xc0014ded40>: failed to create HCP cluster cluster-zstream-4-21-x57jw2: failed waiting for cluster="cluster-zstream-4-21-x57jw2" in resourcegroup="rg-zstream-upgrade-4-21-7ld9jr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7", "name": "706f031f-b30d-4537-9779-a53a5765e7d7", "status": "Failed", "startTime": "2026-04-02T17:28:00.969777194Z", "endTime": "2026-04-02T17:28:53.840813339Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-x57jw2: failed waiting for cluster=\"cluster-zstream-4-21-x57jw2\" in resourcegroup=\"rg-zstream-upgrade-4-21-7ld9jr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"name\": \"706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:28:00.969777194Z\",\n \"endTime\": \"2026-04-02T17:28:53.840813339Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0014ded20>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-x57jw2\" in resourcegroup=\"rg-zstream-upgrade-4-21-7ld9jr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"name\": \"706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:28:00.969777194Z\",\n \"endTime\": \"2026-04-02T17:28:53.840813339Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014cdec0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "95dbdfdf-eccc-4906-8280-98e36991764d", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: A7B02FC3CF694D2B911CFE7230B877F6 Ref B: CH1AA2020617011 Ref C: 2026-04-02T17:29:02Z", ], "Date": [ "Thu, 02 Apr 2026 17:29:02 GMT", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T172902Z:95dbdfdf-eccc-4906-8280-98e36991764d", ], "X-Cache": ["CONFIG_NOCACHE"], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=93... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapError \| 0xc0014ded40>: failed to create HCP cluster cluster-zstream-4-21-x57jw2: failed waiting for cluster="cluster-zstream-4-21-x57jw2" in resourcegroup="rg-zstream-upgrade-4-21-7ld9jr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7", "name": "706f031f-b30d-4537-9779-a53a5765e7d7", "status": "Failed", "startTime": "2026-04-02T17:28:00.969777194Z", "endTime": "2026-04-02T17:28:53.840813339Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-x57jw2: failed waiting for cluster=\"cluster-zstream-4-21-x57jw2\" in resourcegroup=\"rg-zstream-upgrade-4-21-7ld9jr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"name\": \"706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:28:00.969777194Z\",\n \"endTime\": \"2026-04-02T17:28:53.840813339Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0014ded20>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-x57jw2\" in resourcegroup=\"rg-zstream-upgrade-4-21-7ld9jr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"name\": \"706f031f-b30d-4537-9779-a53a5765e7d7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:28:00.969777194Z\",\n \"endTime\": \"2026-04-02T17:28:53.840813339Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014cdec0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "95dbdfdf-eccc-4906-8280-98e36991764d", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: A7B02FC3CF694D2B911CFE7230B877F6 Ref B: CH1AA2020617011 Ref C: 2026-04-02T17:29:02Z", ], "Date": [ "Thu, 02 Apr 2026 17:29:02 GMT", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T172902Z:95dbdfdf-eccc-4906-8280-98e36991764d", ], "X-Cache": ["CONFIG_NOCACHE"], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=93... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 17:14:28 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc0005873e0>: failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster="np-version-upgrade-cluster-qhj6bm" in resourcegroup="rg-np-version-upgrade-qhj6bm-hp2rjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6", "name": "774a88cb-0063-4a09-b407-ee6a7f59a9e6", "status": "Failed", "startTime": "2026-04-02T17:22:52.081012471Z", "endTime": "2026-04-02T17:23:13.48665448Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005873c0>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010284e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["498"], "X-Ms-Correlation-Request-Id": [ "8a74d4be-8a1d-4808-936e-22bae5b4ccd3", ], "X-Msedge-Ref": [ "Ref A: 37CA783E55BC4DE4BA0F59B834B5F490 Ref B: CHI261140504062 Ref C: 2026-04-02T17:23:22Z", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["20"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T172322Z:8a74d4be-8a1d-4808-936e-22bae5b4ccd3", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "c674f25c-3bae-4b96-9f1e... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc0005873e0>: failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster="np-version-upgrade-cluster-qhj6bm" in resourcegroup="rg-np-version-upgrade-qhj6bm-hp2rjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6", "name": "774a88cb-0063-4a09-b407-ee6a7f59a9e6", "status": "Failed", "startTime": "2026-04-02T17:22:52.081012471Z", "endTime": "2026-04-02T17:23:13.48665448Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005873c0>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010284e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["498"], "X-Ms-Correlation-Request-Id": [ "8a74d4be-8a1d-4808-936e-22bae5b4ccd3", ], "X-Msedge-Ref": [ "Ref A: 37CA783E55BC4DE4BA0F59B834B5F490 Ref B: CHI261140504062 Ref C: 2026-04-02T17:23:22Z", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["20"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T172322Z:8a74d4be-8a1d-4808-936e-22bae5b4ccd3", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "c674f25c-3bae-4b96-9f1e... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 17:14:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapError \| 0xc000732640>: failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster="cluster-zstream-4-20-7mzwzp" in resourcegroup="rg-zstream-upgrade-4-20-tq69zx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb", "name": "e3ee4618-26eb-473e-867f-a23c3644e2bb", "status": "Failed", "startTime": "2026-04-02T17:31:57.472030714Z", "endTime": "2026-04-02T17:32:55.049606818Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000732600>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014af200>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/81439e1d-c310-4abd-91e5-f37963acb072", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Length": ["499"], "Content-Type": ["application/json"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 41853D22B093487EB33229F443FF6553 Ref B: CHI261140513025 Ref C: 2026-04-02T17:33:00Z", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapError \| 0xc000732640>: failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster="cluster-zstream-4-20-7mzwzp" in resourcegroup="rg-zstream-upgrade-4-20-tq69zx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb", "name": "e3ee4618-26eb-473e-867f-a23c3644e2bb", "status": "Failed", "startTime": "2026-04-02T17:31:57.472030714Z", "endTime": "2026-04-02T17:32:55.049606818Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000732600>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014af200>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/81439e1d-c310-4abd-91e5-f37963acb072", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Length": ["499"], "Content-Type": ["application/json"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 41853D22B093487EB33229F443FF6553 Ref B: CHI261140513025 Ref C: 2026-04-02T17:33:00Z", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
08:48:43	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	8	e2e (8) ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc000e5d680>: failed to create HCP cluster np-version-upgrade-cluster-zlcbwt: failed waiting for cluster="np-version-upgrade-cluster-zlcbwt" in resourcegroup="rg-np-version-upgrade-zlcbwt-xdw7n4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c", "name": "f6a47a17-e228-4123-8880-e17806ecd15c", "status": "Failed", "startTime": "2026-04-02T09:00:13.820056162Z", "endTime": "2026-04-02T09:00:48.215590338Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-zlcbwt: failed waiting for cluster=\"np-version-upgrade-cluster-zlcbwt\" in resourcegroup=\"rg-np-version-upgrade-zlcbwt-xdw7n4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"name\": \"f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:00:13.820056162Z\",\n \"endTime\": \"2026-04-02T09:00:48.215590338Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e5d660>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-zlcbwt\" in resourcegroup=\"rg-np-version-upgrade-zlcbwt-xdw7n4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"name\": \"f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:00:13.820056162Z\",\n \"endTime\": \"2026-04-02T09:00:48.215590338Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00114aba0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T090055Z:9658587b-26d3-458b-83e1-b589efdf4111", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Thu, 02 Apr 2026 09:00:54 GMT", ], "X-Ms-Correlation-Request-Id": [ "9658587b-26d3-458b-83e1-b589efdf4111", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Content-Length": ["499"], "X-Ms-Request-Id": [ "de87e374-c0ba-41f3-b4e8-34237314bf28", ], "X-Envoy-Upstream-Service-Time": ["23"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,obje... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc000e5d680>: failed to create HCP cluster np-version-upgrade-cluster-zlcbwt: failed waiting for cluster="np-version-upgrade-cluster-zlcbwt" in resourcegroup="rg-np-version-upgrade-zlcbwt-xdw7n4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c", "name": "f6a47a17-e228-4123-8880-e17806ecd15c", "status": "Failed", "startTime": "2026-04-02T09:00:13.820056162Z", "endTime": "2026-04-02T09:00:48.215590338Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-zlcbwt: failed waiting for cluster=\"np-version-upgrade-cluster-zlcbwt\" in resourcegroup=\"rg-np-version-upgrade-zlcbwt-xdw7n4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"name\": \"f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:00:13.820056162Z\",\n \"endTime\": \"2026-04-02T09:00:48.215590338Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e5d660>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-zlcbwt\" in resourcegroup=\"rg-np-version-upgrade-zlcbwt-xdw7n4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"name\": \"f6a47a17-e228-4123-8880-e17806ecd15c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:00:13.820056162Z\",\n \"endTime\": \"2026-04-02T09:00:48.215590338Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00114aba0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260402T090055Z:9658587b-26d3-458b-83e1-b589efdf4111", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Thu, 02 Apr 2026 09:00:54 GMT", ], "X-Ms-Correlation-Request-Id": [ "9658587b-26d3-458b-83e1-b589efdf4111", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Content-Length": ["499"], "X-Ms-Request-Id": [ "de87e374-c0ba-41f3-b4e8-34237314bf28", ], "X-Envoy-Upstream-Service-Time": ["23"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,obje... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc000d04420>: failed to create HCP cluster np-version-upgrade-cluster-fd27fq: failed waiting for cluster="np-version-upgrade-cluster-fd27fq" in resourcegroup="rg-np-version-upgrade-fd27fq-kmzsnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446", "name": "faaed912-8c9c-4b9c-a560-ae5d271fe446", "status": "Failed", "startTime": "2026-04-02T09:03:17.642456763Z", "endTime": "2026-04-02T09:03:33.609008884Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-fd27fq: failed waiting for cluster=\"np-version-upgrade-cluster-fd27fq\" in resourcegroup=\"rg-np-version-upgrade-fd27fq-kmzsnf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"name\": \"faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:17.642456763Z\",\n \"endTime\": \"2026-04-02T09:03:33.609008884Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d04400>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-fd27fq\" in resourcegroup=\"rg-np-version-upgrade-fd27fq-kmzsnf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"name\": \"faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:17.642456763Z\",\n \"endTime\": \"2026-04-02T09:03:33.609008884Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00142c780>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "Pragma": ["no-cache"], "Content-Length": ["499"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "22fdfd66-2ef7-48dc-92f9-15957d9814a7", ], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260402T090338Z:b2299bf9-2580-4e04-bcf6-90257fc5e438", ], "Cache-Control": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/bc50bcf7-9fe7-40b0-8517-584011411f7c", ], "X-Ms-Correlation-Request-Id": [ "b2299bf9-2580-4e04-bcf6-90257fc5e438", ], "Strict-Transport-Security": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error: <fmt.wrapError \| 0xc000d04420>: failed to create HCP cluster np-version-upgrade-cluster-fd27fq: failed waiting for cluster="np-version-upgrade-cluster-fd27fq" in resourcegroup="rg-np-version-upgrade-fd27fq-kmzsnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446", "name": "faaed912-8c9c-4b9c-a560-ae5d271fe446", "status": "Failed", "startTime": "2026-04-02T09:03:17.642456763Z", "endTime": "2026-04-02T09:03:33.609008884Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-version-upgrade-cluster-fd27fq: failed waiting for cluster=\"np-version-upgrade-cluster-fd27fq\" in resourcegroup=\"rg-np-version-upgrade-fd27fq-kmzsnf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"name\": \"faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:17.642456763Z\",\n \"endTime\": \"2026-04-02T09:03:33.609008884Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d04400>{ msg: "failed waiting for cluster=\"np-version-upgrade-cluster-fd27fq\" in resourcegroup=\"rg-np-version-upgrade-fd27fq-kmzsnf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"name\": \"faaed912-8c9c-4b9c-a560-ae5d271fe446\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:17.642456763Z\",\n \"endTime\": \"2026-04-02T09:03:33.609008884Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00142c780>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "Pragma": ["no-cache"], "Content-Length": ["499"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "22fdfd66-2ef7-48dc-92f9-15957d9814a7", ], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260402T090338Z:b2299bf9-2580-4e04-bcf6-90257fc5e438", ], "Cache-Control": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/bc50bcf7-9fe7-40b0-8517-584011411f7c", ], "X-Ms-Correlation-Request-Id": [ "b2299bf9-2580-4e04-bcf6-90257fc5e438", ], "Strict-Transport-Security": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapError \| 0xc000880260>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-6tff8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: ServiceUnavailable -------------------------------------------------------------------------------- { "error": { "code": "ServiceUnavailable", "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-cluster: failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000880200>{ msg: "failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012ca1b0>{ ErrorCode: "ServiceUnavailable", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "2a87bdad-8340-4cda-a740-2d7cc278c733", ], "X-Ms-Correlation-Request-Id": [ "294bd1e5-7abf-4f05-a3c2-931fa8a72c4e", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T085750Z:294bd1e5-7abf-4f05-a3c2-931fa8a72c4e", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Date": [ "Thu, 02 Apr 2026 08:57:50 GMT", ], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["2649"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/cc9e8570-fcc2-4ba9-9d49-08bb67825537", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Content-Type-Options": ["nosniff"], "Content-Length": ["257"], "Retry-After": ["59"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Fail... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapError \| 0xc000880260>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-6tff8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: ServiceUnavailable -------------------------------------------------------------------------------- { "error": { "code": "ServiceUnavailable", "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-cluster: failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000880200>{ msg: "failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012ca1b0>{ ErrorCode: "ServiceUnavailable", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "2a87bdad-8340-4cda-a740-2d7cc278c733", ], "X-Ms-Correlation-Request-Id": [ "294bd1e5-7abf-4f05-a3c2-931fa8a72c4e", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T085750Z:294bd1e5-7abf-4f05-a3c2-931fa8a72c4e", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Date": [ "Thu, 02 Apr 2026 08:57:50 GMT", ], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["2649"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/cc9e8570-fcc2-4ba9-9d49-08bb67825537", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Content-Type-Options": ["nosniff"], "Content-Length": ["257"], "Retry-After": ["59"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Fail... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error: <fmt.wrapError \| 0xc000a2f2c0>: failed to create HCP cluster cluster-zstream-4-20-b88v27: failed waiting for cluster="cluster-zstream-4-20-b88v27" in resourcegroup="rg-zstream-upgrade-4-20-lmrfzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923", "name": "d0e3a763-ff68-4aa2-acc4-15b05e869923", "status": "Failed", "startTime": "2026-04-02T09:02:34.406522828Z", "endTime": "2026-04-02T09:02:43.014191999Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-b88v27: failed waiting for cluster=\"cluster-zstream-4-20-b88v27\" in resourcegroup=\"rg-zstream-upgrade-4-20-lmrfzn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"name\": \"d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:34.406522828Z\",\n \"endTime\": \"2026-04-02T09:02:43.014191999Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a2f2a0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-b88v27\" in resourcegroup=\"rg-zstream-upgrade-4-20-lmrfzn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"name\": \"d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:34.406522828Z\",\n \"endTime\": \"2026-04-02T09:02:43.014191999Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008428a0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260402T090245Z:f15f4ae6-036f-4104-9d3c-5b007bb48b61", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Content-Length": ["499"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/b395eb27-1a6a-4aac-b717-f00f205f4a7c", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error: <fmt.wrapError \| 0xc000a2f2c0>: failed to create HCP cluster cluster-zstream-4-20-b88v27: failed waiting for cluster="cluster-zstream-4-20-b88v27" in resourcegroup="rg-zstream-upgrade-4-20-lmrfzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923", "name": "d0e3a763-ff68-4aa2-acc4-15b05e869923", "status": "Failed", "startTime": "2026-04-02T09:02:34.406522828Z", "endTime": "2026-04-02T09:02:43.014191999Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-b88v27: failed waiting for cluster=\"cluster-zstream-4-20-b88v27\" in resourcegroup=\"rg-zstream-upgrade-4-20-lmrfzn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"name\": \"d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:34.406522828Z\",\n \"endTime\": \"2026-04-02T09:02:43.014191999Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a2f2a0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-b88v27\" in resourcegroup=\"rg-zstream-upgrade-4-20-lmrfzn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"name\": \"d0e3a763-ff68-4aa2-acc4-15b05e869923\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:34.406522828Z\",\n \"endTime\": \"2026-04-02T09:02:43.014191999Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008428a0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260402T090245Z:f15f4ae6-036f-4104-9d3c-5b007bb48b61", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Content-Length": ["499"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/b395eb27-1a6a-4aac-b717-f00f205f4a7c", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error: <fmt.wrapError \| 0xc000b42240>: failed to create HCP cluster cluster-zstream-4-19-vq7bzx: failed waiting for cluster="cluster-zstream-4-19-vq7bzx" in resourcegroup="rg-zstream-upgrade-4-19-rjrsjm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1", "name": "d9376b46-84bf-4503-b92d-26a4202504a1", "status": "Failed", "startTime": "2026-04-02T09:03:10.501037893Z", "endTime": "2026-04-02T09:03:33.384537422Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-vq7bzx: failed waiting for cluster=\"cluster-zstream-4-19-vq7bzx\" in resourcegroup=\"rg-zstream-upgrade-4-19-rjrsjm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"name\": \"d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:10.501037893Z\",\n \"endTime\": \"2026-04-02T09:03:33.384537422Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b42220>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-vq7bzx\" in resourcegroup=\"rg-zstream-upgrade-4-19-rjrsjm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"name\": \"d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:10.501037893Z\",\n \"endTime\": \"2026-04-02T09:03:33.384537422Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000fba6f0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Msedge-Ref": [ "Ref A: 76C9DB92CECA4A7F9DE40C0EAE9E1D56 Ref B: CHI261140512052 Ref C: 2026-04-02T09:03:41Z", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "1422b14c-34d2-442e-88de-df672336bbff", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Envoy-Upstream-Service-Time": ["18"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/b1a63070-9329-4e22-b9b2-867d3a3e07c9", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Re... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error: <fmt.wrapError \| 0xc000b42240>: failed to create HCP cluster cluster-zstream-4-19-vq7bzx: failed waiting for cluster="cluster-zstream-4-19-vq7bzx" in resourcegroup="rg-zstream-upgrade-4-19-rjrsjm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1", "name": "d9376b46-84bf-4503-b92d-26a4202504a1", "status": "Failed", "startTime": "2026-04-02T09:03:10.501037893Z", "endTime": "2026-04-02T09:03:33.384537422Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-vq7bzx: failed waiting for cluster=\"cluster-zstream-4-19-vq7bzx\" in resourcegroup=\"rg-zstream-upgrade-4-19-rjrsjm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"name\": \"d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:10.501037893Z\",\n \"endTime\": \"2026-04-02T09:03:33.384537422Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b42220>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-vq7bzx\" in resourcegroup=\"rg-zstream-upgrade-4-19-rjrsjm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"name\": \"d9376b46-84bf-4503-b92d-26a4202504a1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:03:10.501037893Z\",\n \"endTime\": \"2026-04-02T09:03:33.384537422Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000fba6f0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Msedge-Ref": [ "Ref A: 76C9DB92CECA4A7F9DE40C0EAE9E1D56 Ref B: CHI261140512052 Ref C: 2026-04-02T09:03:41Z", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "1422b14c-34d2-442e-88de-df672336bbff", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Envoy-Upstream-Service-Time": ["18"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/b1a63070-9329-4e22-b9b2-867d3a3e07c9", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Re... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapError \| 0xc00075c1e0>: failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-wvg7xj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: ServiceUnavailable -------------------------------------------------------------------------------- { "error": { "code": "ServiceUnavailable", "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-name-cluster: failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00075c1c0>{ msg: "failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00004d2f0>{ ErrorCode: "ServiceUnavailable", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["2878"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Correlation-Request-Id": [ "0b9741c0-9777-4781-b4fb-bf1afce41505", ], "Retry-After": ["59"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Date": [ "Thu, 02 Apr 2026 08:57:47 GMT", ], "Content-Length": ["257"], "Content-Type": ["application/json"], "X-Ms-Error-Code": ["ServiceUnavailable"], "X-Ms-Request-Id": [ "87531355-f28f-4c12-bf8b-c7f8a31db6ce", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/7ad18b62-8874-4d58-81d6-01921cd42fc7", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T085747Z:0b9741c0-9777-4781-b4fb-bf1afce41505", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Expires": ["-1"], "X-Ms-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapError \| 0xc00075c1e0>: failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-wvg7xj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: ServiceUnavailable -------------------------------------------------------------------------------- { "error": { "code": "ServiceUnavailable", "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-name-cluster: failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00075c1c0>{ msg: "failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00004d2f0>{ ErrorCode: "ServiceUnavailable", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["2878"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Correlation-Request-Id": [ "0b9741c0-9777-4781-b4fb-bf1afce41505", ], "Retry-After": ["59"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Date": [ "Thu, 02 Apr 2026 08:57:47 GMT", ], "Content-Length": ["257"], "Content-Type": ["application/json"], "X-Ms-Error-Code": ["ServiceUnavailable"], "X-Ms-Request-Id": [ "87531355-f28f-4c12-bf8b-c7f8a31db6ce", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/7ad18b62-8874-4d58-81d6-01921cd42fc7", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T085747Z:0b9741c0-9777-4781-b4fb-bf1afce41505", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Expires": ["-1"], "X-Ms-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:123]: Unexpected error: <fmt.wrapError \| 0xc0004e6700>: failed starting cluster creation "basic-hcp-cluster2" in resourcegroup="customer-rg-p6dr85": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2 -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: ServiceUnavailable -------------------------------------------------------------------------------- { "error": { "code": "ServiceUnavailable", "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later" } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-p6dr85\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001402630>{ ErrorCode: "ServiceUnavailable", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "981e3b54-276e-45f0-a48f-b05f015c79dc", ], "X-Envoy-Upstream-Service-Time": ["8461"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/d750dcff-9cd4-44ec-8a65-6f1b69e57a9d", ], "X-Ms-Correlation-Request-Id": [ "45bdda71-4b5b-4b9b-9fd7-4df364bf7808", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T085811Z:45bdda71-4b5b-4b9b-9fd7-4df364bf7808", ], "Pragma": ["no-cache"], "Content-Length": ["257"], "Expires": ["-1"], "X-Ms-Error-Code": ["ServiceUnavailable"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: C36632182EBE4B8EB52F9106BF8C5BB2 Ref B: CHI261140502029 Ref C: 2026-04-02T08:58:02Z", ], "Retry-After": ["59"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Date": [ "Thu, 02 Apr 2026 08:58:11 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0004e6480>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 83, 101, 114, 118, 105, 99, 101, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 105, 110, 115, 116, 97, 108, 108, 97, 116, 105, 111, 110, 32, 104, 97, 115, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 99, 97, 112, 97, 99, 105, 116, 121, 46, 32, 87, 104, 105, 108, 101, 32, 119, 101, 32, 97, 114, 101, 32, 98, 114, 105, 110, 103, 105, 110, 103, 32, 111, 110, 32, 97, 100, 100, 105, 116, 105, 111, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:123]: Unexpected error: <fmt.wrapError \| 0xc0004e6700>: failed starting cluster creation "basic-hcp-cluster2" in resourcegroup="customer-rg-p6dr85": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2 -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: ServiceUnavailable -------------------------------------------------------------------------------- { "error": { "code": "ServiceUnavailable", "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later" } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-p6dr85\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001402630>{ ErrorCode: "ServiceUnavailable", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "981e3b54-276e-45f0-a48f-b05f015c79dc", ], "X-Envoy-Upstream-Service-Time": ["8461"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/d750dcff-9cd4-44ec-8a65-6f1b69e57a9d", ], "X-Ms-Correlation-Request-Id": [ "45bdda71-4b5b-4b9b-9fd7-4df364bf7808", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T085811Z:45bdda71-4b5b-4b9b-9fd7-4df364bf7808", ], "Pragma": ["no-cache"], "Content-Length": ["257"], "Expires": ["-1"], "X-Ms-Error-Code": ["ServiceUnavailable"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: C36632182EBE4B8EB52F9106BF8C5BB2 Ref B: CHI261140502029 Ref C: 2026-04-02T08:58:02Z", ], "Retry-After": ["59"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Date": [ "Thu, 02 Apr 2026 08:58:11 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0004e6480>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 83, 101, 114, 118, 105, 99, 101, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 105, 110, 115, 116, 97, 108, 108, 97, 116, 105, 111, 110, 32, 104, 97, 115, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 99, 97, 112, 97, 99, 105, 116, 121, 46, 32, 87, 104, 105, 108, 101, 32, 119, 101, 32, 97, 114, 101, 32, 98, 114, 105, 110, 103, 105, 110, 103, 32, 111, 110, 32, 97, 100, 100, 105, 116, 105, 111, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 08:48:43 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error: <fmt.wrapError \| 0xc0007b6ba0>: failed to create HCP cluster cluster-zstream-4-21-59tjhl: failed waiting for cluster="cluster-zstream-4-21-59tjhl" in resourcegroup="rg-zstream-upgrade-4-21-rddnsg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec", "name": "bab93702-2495-45ca-bdfe-fef91e9947ec", "status": "Failed", "startTime": "2026-04-02T09:02:38.689735931Z", "endTime": "2026-04-02T09:02:55.541508676Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-59tjhl: failed waiting for cluster=\"cluster-zstream-4-21-59tjhl\" in resourcegroup=\"rg-zstream-upgrade-4-21-rddnsg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"name\": \"bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:38.689735931Z\",\n \"endTime\": \"2026-04-02T09:02:55.541508676Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007b6b00>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-59tjhl\" in resourcegroup=\"rg-zstream-upgrade-4-21-rddnsg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"name\": \"bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:38.689735931Z\",\n \"endTime\": \"2026-04-02T09:02:55.541508676Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000f46930>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Content-Length": ["499"], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "X-Ms-Request-Id": [ "c629973c-9d73-46a2-a7e1-a7d37a1a5a9a", ], "X-Envoy-Upstream-Service-Time": ["41"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/e829b8ea-8772-4c59-8a5b-1fa05e5027c2", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Msedge-Ref": [ "Ref A: F83DD0A50C8A422096F238C28BB336A7 Ref B: CHI261140513031 Ref C: 2026-04-02T09:02:59Z", ], "Expire... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error: <fmt.wrapError \| 0xc0007b6ba0>: failed to create HCP cluster cluster-zstream-4-21-59tjhl: failed waiting for cluster="cluster-zstream-4-21-59tjhl" in resourcegroup="rg-zstream-upgrade-4-21-rddnsg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec", "name": "bab93702-2495-45ca-bdfe-fef91e9947ec", "status": "Failed", "startTime": "2026-04-02T09:02:38.689735931Z", "endTime": "2026-04-02T09:02:55.541508676Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-59tjhl: failed waiting for cluster=\"cluster-zstream-4-21-59tjhl\" in resourcegroup=\"rg-zstream-upgrade-4-21-rddnsg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"name\": \"bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:38.689735931Z\",\n \"endTime\": \"2026-04-02T09:02:55.541508676Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007b6b00>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-59tjhl\" in resourcegroup=\"rg-zstream-upgrade-4-21-rddnsg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"name\": \"bab93702-2495-45ca-bdfe-fef91e9947ec\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T09:02:38.689735931Z\",\n \"endTime\": \"2026-04-02T09:02:55.541508676Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000f46930>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Content-Length": ["499"], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "X-Ms-Request-Id": [ "c629973c-9d73-46a2-a7e1-a7d37a1a5a9a", ], "X-Envoy-Upstream-Service-Time": ["41"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/e829b8ea-8772-4c59-8a5b-1fa05e5027c2", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Msedge-Ref": [ "Ref A: F83DD0A50C8A422096F238C28BB336A7 Ref B: CHI261140513031 Ref C: 2026-04-02T09:02:59Z", ], "Expire... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
05:27:45	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
01:37:08	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	12	e2e (12) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error: <fmt.wrapError \| 0xc001418640>: failed creating external auth "ea-list" in resourcegroup="ea-list-sqv9wn" for cluster="ea-list4lvwkr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-sqv9wn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list4lvwkr/externalAuths/ea-list -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-sqv9wn\" for cluster=\"ea-list4lvwkr\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-sqv9wn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list4lvwkr/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001366990>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Request-Id": [ "f4e61ff2-b02c-4f78-930a-516f5367173c", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Date": [ "Thu, 02 Apr 2026 02:10:05 GMT", ], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 04AE8E5AF86B478685A5B528872AB82B Ref B: CHI261140505062 Ref C: 2026-04-02T02:09:59Z", ], "Content-Length": ["107"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/786cdfbd-1a09-42c1-be7d-896a3083330d", ], "X-Ms-Correlation-Request-Id": [ "4be39819-497a-401e-a7b1-442d6a3bb89a", ], "X-Envoy-Upstream-Service-Time": ["7051"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021006Z:4be39819-497a-401e-a7b1-442d6a3bb89a", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc001418500>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error: <fmt.wrapError \| 0xc001418640>: failed creating external auth "ea-list" in resourcegroup="ea-list-sqv9wn" for cluster="ea-list4lvwkr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-sqv9wn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list4lvwkr/externalAuths/ea-list -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-sqv9wn\" for cluster=\"ea-list4lvwkr\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-sqv9wn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list4lvwkr/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001366990>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Request-Id": [ "f4e61ff2-b02c-4f78-930a-516f5367173c", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Date": [ "Thu, 02 Apr 2026 02:10:05 GMT", ], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 04AE8E5AF86B478685A5B528872AB82B Ref B: CHI261140505062 Ref C: 2026-04-02T02:09:59Z", ], "Content-Length": ["107"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/786cdfbd-1a09-42c1-be7d-896a3083330d", ], "X-Ms-Correlation-Request-Id": [ "4be39819-497a-401e-a7b1-442d6a3bb89a", ], "X-Envoy-Upstream-Service-Time": ["7051"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021006Z:4be39819-497a-401e-a7b1-442d6a3bb89a", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc001418500>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 01:37:08 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc0017d0f90>: failed waiting for hcpCluster="admin-cred-lifecycle-lvdt77" in resourcegroup="admin-credential-lifecycle-test-cdl8dq" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-lvdt77 in resource group admin-credential-lifecycle-test-cdl8dq, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-lvdt77\" in resourcegroup=\"admin-credential-lifecycle-test-cdl8dq\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-lvdt77 in resource group admin-credential-lifecycle-test-cdl8dq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0000d0170>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-lvdt77 in resource group admin-credential-lifecycle-test-cdl8dq", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc0017d0f90>: failed waiting for hcpCluster="admin-cred-lifecycle-lvdt77" in resourcegroup="admin-credential-lifecycle-test-cdl8dq" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-lvdt77 in resource group admin-credential-lifecycle-test-cdl8dq, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-lvdt77\" in resourcegroup=\"admin-credential-lifecycle-test-cdl8dq\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-lvdt77 in resource group admin-credential-lifecycle-test-cdl8dq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0000d0170>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-lvdt77 in resource group admin-credential-lifecycle-test-cdl8dq", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapError \| 0xc00142ee20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00104d530>{ ErrorCode: "LockContention", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Content-Type-Options": ["nosniff"], "Content-Type": ["application/json"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/0949565f-2116-4c2d-ac42-f874e16826f1", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021009Z:c60ff00c-0498-471e-83f9-643c3970b3df", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 84A2B7061A1D427CAC14B9844929E76E Ref B: CHI261140502054 Ref C: 2026-04-02T02:09:58Z", ], "Pragma": ["no-cache"], "Content-Length": ["186"], "Retry-After": ["10"], "X-Ms-Error-Code": ["LockContention"], "X-Ms-Request-Id": [ "342b364f-60a1-493c-a980-5f75e6f1c77f", ], "X-Envoy-Upstream-Service-Time": ["10010"], "Date": [ "Thu, 02 Apr 2026 02:10:08 GMT", ], "Cache-Control": ["no-cache"], "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "c60ff00c-0498-471e-83f9-643c3970b3df", ], }, Body: <exported.nopClosingBytesReader \| 0xc00142ed60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, Co... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapError \| 0xc00142ee20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00104d530>{ ErrorCode: "LockContention", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Content-Type-Options": ["nosniff"], "Content-Type": ["application/json"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/0949565f-2116-4c2d-ac42-f874e16826f1", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021009Z:c60ff00c-0498-471e-83f9-643c3970b3df", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 84A2B7061A1D427CAC14B9844929E76E Ref B: CHI261140502054 Ref C: 2026-04-02T02:09:58Z", ], "Pragma": ["no-cache"], "Content-Length": ["186"], "Retry-After": ["10"], "X-Ms-Error-Code": ["LockContention"], "X-Ms-Request-Id": [ "342b364f-60a1-493c-a980-5f75e6f1c77f", ], "X-Envoy-Upstream-Service-Time": ["10010"], "Date": [ "Thu, 02 Apr 2026 02:10:08 GMT", ], "Cache-Control": ["no-cache"], "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "c60ff00c-0498-471e-83f9-643c3970b3df", ], }, Body: <exported.nopClosingBytesReader \| 0xc00142ed60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, Co... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error: <fmt.wrapError \| 0xc000a814e0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-n7b9pd-pbv84h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-n7b9pd/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-n7b9pd-pbv84h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-n7b9pd/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bd97a0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022003Z:0a43673e-6459-4a7a-9684-1ffc756dffcb", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 4F8CAF08C1B3477AAEC3FF6E1D6910E3 Ref B: CHI261140505040 Ref C: 2026-04-02T02:20:02Z", ], "Date": [ "Thu, 02 Apr 2026 02:20:02 GMT", ], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["663"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/91a17ebb-245e-4f3f-89d3-25fb7c6bd722", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Content-Length": ["107"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "66ee99e3-1f96-460f-b64b-d03b134f3ec4", ], "X-Ms-Failure-Cause": ["service"], "Cache-Control": ["no-cache"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Correlation-Request-Id": [ "0a43673e-6459-4a7a-9684-1ffc756dffcb", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a81420>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error: <fmt.wrapError \| 0xc000a814e0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-n7b9pd-pbv84h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-n7b9pd/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-n7b9pd-pbv84h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-n7b9pd/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bd97a0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022003Z:0a43673e-6459-4a7a-9684-1ffc756dffcb", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 4F8CAF08C1B3477AAEC3FF6E1D6910E3 Ref B: CHI261140505040 Ref C: 2026-04-02T02:20:02Z", ], "Date": [ "Thu, 02 Apr 2026 02:20:02 GMT", ], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["663"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/91a17ebb-245e-4f3f-89d3-25fb7c6bd722", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Content-Length": ["107"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "66ee99e3-1f96-460f-b64b-d03b134f3ec4", ], "X-Ms-Failure-Cause": ["service"], "Cache-Control": ["no-cache"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Correlation-Request-Id": [ "0a43673e-6459-4a7a-9684-1ffc756dffcb", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a81420>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:147]: Unexpected error: <fmt.wrapError \| 0xc0008c23a0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-c4ph7r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pw4hx4/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-c4ph7r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pw4hx4/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00031b110>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022005Z:bae2f98c-c749-481d-9054-9921e081945e", ], "X-Content-Type-Options": ["nosniff"], "Pragma": ["no-cache"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/62d919b4-4225-4967-9df6-08b0b5351023", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Thu, 02 Apr 2026 02:20:05 GMT", ], "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "4c5f9568-bb9b-40c3-81ad-0ca630f99d24", ], "X-Msedge-Ref": [ "Ref A: 2552FC52616C4D5BB46364A7B7C0692E Ref B: CH1AA2020617017 Ref C: 2026-04-02T02:20:02Z", ], "X-Envoy-Upstream-Service-Time": ["2628"], "X-Ms-Failure-Cause": ["service"], "Content-Length": ["107"], "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "bae2f98c-c749-481d-9054-9921e081945e", ], }, Body: <exported.nopClosingBytesReader \| 0xc0008c21e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:147]: Unexpected error: <fmt.wrapError \| 0xc0008c23a0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-c4ph7r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pw4hx4/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-c4ph7r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pw4hx4/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00031b110>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022005Z:bae2f98c-c749-481d-9054-9921e081945e", ], "X-Content-Type-Options": ["nosniff"], "Pragma": ["no-cache"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/62d919b4-4225-4967-9df6-08b0b5351023", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Thu, 02 Apr 2026 02:20:05 GMT", ], "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "4c5f9568-bb9b-40c3-81ad-0ca630f99d24", ], "X-Msedge-Ref": [ "Ref A: 2552FC52616C4D5BB46364A7B7C0692E Ref B: CH1AA2020617017 Ref C: 2026-04-02T02:20:02Z", ], "X-Envoy-Upstream-Service-Time": ["2628"], "X-Ms-Failure-Cause": ["service"], "Content-Length": ["107"], "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "bae2f98c-c749-481d-9054-9921e081945e", ], }, Body: <exported.nopClosingBytesReader \| 0xc0008c21e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 01:37:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc000f5e7b0>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-qcjg4q" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qcjg4q, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qcjg4q\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qcjg4q, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015ab170>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qcjg4q", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc000f5e7b0>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-qcjg4q" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qcjg4q, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qcjg4q\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qcjg4q, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015ab170>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qcjg4q", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error: <fmt.wrapError \| 0xc00080bb60>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-h6h9kt-6zms4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-h6h9kt/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-h6h9kt-6zms4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-h6h9kt/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000792300>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Content-Length": ["107"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["339"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/db8b7c42-a699-4749-a2d0-f6d8f859aacd", ], "X-Ms-Correlation-Request-Id": [ "35b54f7f-739e-4203-895d-a04b57b209f1", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Ms-Request-Id": [ "dd3c2403-c919-40cc-afa1-b2bd95e920f7", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 8A45806B62A847B0A86E8C8C1276610B Ref B: CHI261140508031 Ref C: 2026-04-02T02:24:34Z", ], "Date": [ "Thu, 02 Apr 2026 02:24:34 GMT", ], "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022434Z:35b54f7f-739e-4203-895d-a04b57b209f1", ], "X-Cache": ["CONFIG_NOCACHE"], "Pragma": ["no-cache"], "X-Ms-Error-Code": [ "InternalServerError", ], }, Body: <exported.nopClosingBytesReader \| 0xc00080b860>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error: <fmt.wrapError \| 0xc00080bb60>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-h6h9kt-6zms4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-h6h9kt/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-h6h9kt-6zms4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-h6h9kt/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000792300>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Content-Length": ["107"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["339"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/db8b7c42-a699-4749-a2d0-f6d8f859aacd", ], "X-Ms-Correlation-Request-Id": [ "35b54f7f-739e-4203-895d-a04b57b209f1", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Ms-Request-Id": [ "dd3c2403-c919-40cc-afa1-b2bd95e920f7", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 8A45806B62A847B0A86E8C8C1276610B Ref B: CHI261140508031 Ref C: 2026-04-02T02:24:34Z", ], "Date": [ "Thu, 02 Apr 2026 02:24:34 GMT", ], "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022434Z:35b54f7f-739e-4203-895d-a04b57b209f1", ], "X-Cache": ["CONFIG_NOCACHE"], "Pragma": ["no-cache"], "X-Ms-Error-Code": [ "InternalServerError", ], }, Body: <exported.nopClosingBytesReader \| 0xc00080b860>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapError \| 0xc0009f2840>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00159e990>{ ErrorCode: "LockContention", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1c5d69eb-b218-43c3-89e1-797546221c53", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Thu, 02 Apr 2026 02:10:08 GMT", ], "Content-Type": ["application/json"], "X-Ms-Error-Code": ["LockContention"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021008Z:afdf4447-88c0-4010-85eb-2250eadd30fe", ], "X-Msedge-Ref": [ "Ref A: 1EC287F8926644E099B85CCB3B5E9590 Ref B: CHI261140504042 Ref C: 2026-04-02T02:09:58Z", ], "Content-Length": ["186"], "Expires": ["-1"], "X-Ms-Request-Id": [ "2b953333-0bff-468f-ad05-d66baee5ebf5", ], "X-Envoy-Upstream-Service-Time": ["10006"], "X-Ms-Correlation-Request-Id": [ "afdf4447-88c0-4010-85eb-2250eadd30fe", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Retry-After": ["10"], "X-Ms-Failure-Cause": ["service"], }, Body: <exported.nopClosingBytesReader \| 0xc0009f2700>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, Conte... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapError \| 0xc0009f2840>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00159e990>{ ErrorCode: "LockContention", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1c5d69eb-b218-43c3-89e1-797546221c53", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Thu, 02 Apr 2026 02:10:08 GMT", ], "Content-Type": ["application/json"], "X-Ms-Error-Code": ["LockContention"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021008Z:afdf4447-88c0-4010-85eb-2250eadd30fe", ], "X-Msedge-Ref": [ "Ref A: 1EC287F8926644E099B85CCB3B5E9590 Ref B: CHI261140504042 Ref C: 2026-04-02T02:09:58Z", ], "Content-Length": ["186"], "Expires": ["-1"], "X-Ms-Request-Id": [ "2b953333-0bff-468f-ad05-d66baee5ebf5", ], "X-Envoy-Upstream-Service-Time": ["10006"], "X-Ms-Correlation-Request-Id": [ "afdf4447-88c0-4010-85eb-2250eadd30fe", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Retry-After": ["10"], "X-Ms-Failure-Cause": ["service"], }, Body: <exported.nopClosingBytesReader \| 0xc0009f2700>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, Conte... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 01:37:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:106]: Unexpected error: <fmt.wrapErrors \| 0xc0011440f0>: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-zzfq7g" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-zzfq7g, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-zzfq7g\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-zzfq7g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000ef19b0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-zzfq7g", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:106]: Unexpected error: <fmt.wrapErrors \| 0xc0011440f0>: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-zzfq7g" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-zzfq7g, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-zzfq7g\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-zzfq7g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000ef19b0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-zzfq7g", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 01:37:08 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc0009b8fc0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-7ntzdq" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-7ntzdq, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-7ntzdq\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-7ntzdq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000676740>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-7ntzdq", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc0009b8fc0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-7ntzdq" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-7ntzdq, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-7ntzdq\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-7ntzdq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000676740>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-7ntzdq", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error: <fmt.wrapError \| 0xc0012b0780>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-8kqbht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-8kqbht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012f34a0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Content-Length": ["107"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1f894e75-bf9b-4f38-b36f-086c2d1ebd50", ], "X-Msedge-Ref": [ "Ref A: 49D6E7E0AC224A1688162351432A0445 Ref B: CH1AA2020610019 Ref C: 2026-04-02T02:19:06Z", ], "Date": [ "Thu, 02 Apr 2026 02:19:06 GMT", ], "X-Ms-Request-Id": [ "4c158d49-d82a-452b-867b-1015459e1703", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021907Z:0ff2a06f-4992-4cf9-ae72-86b939159372", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["399"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Correlation-Request-Id": [ "0ff2a06f-4992-4cf9-ae72-86b939159372", ], }, Body: <exported.nopClosingBytesReader \| 0xc0012b06c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "manageme... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error: <fmt.wrapError \| 0xc0012b0780>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-8kqbht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-8kqbht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012f34a0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Content-Length": ["107"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1f894e75-bf9b-4f38-b36f-086c2d1ebd50", ], "X-Msedge-Ref": [ "Ref A: 49D6E7E0AC224A1688162351432A0445 Ref B: CH1AA2020610019 Ref C: 2026-04-02T02:19:06Z", ], "Date": [ "Thu, 02 Apr 2026 02:19:06 GMT", ], "X-Ms-Request-Id": [ "4c158d49-d82a-452b-867b-1015459e1703", ], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T021907Z:0ff2a06f-4992-4cf9-ae72-86b939159372", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["399"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Correlation-Request-Id": [ "0ff2a06f-4992-4cf9-ae72-86b939159372", ], }, Body: <exported.nopClosingBytesReader \| 0xc0012b06c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "manageme... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 01:37:08 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:192]: Unexpected error: <fmt.wrapError \| 0xc0011d2100>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-t8jz2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-t8jz2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00086f3e0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/58e1c38c-4da4-44dd-994d-8cbc32ac06f7", ], "X-Ms-Correlation-Request-Id": [ "3536f45d-afc7-473f-8b37-bad9de7684d7", ], "Date": [ "Thu, 02 Apr 2026 02:22:32 GMT", ], "Content-Type": ["application/json"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: F4CED9AE58F244F885602E16952E2D6D Ref B: CH1AA2020617037 Ref C: 2026-04-02T02:22:31Z", ], "X-Ms-Request-Id": [ "f2afc4a3-ed79-4022-adac-dc064151bd9a", ], "X-Envoy-Upstream-Service-Time": ["744"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Cache": ["CONFIG_NOCACHE"], "Expires": ["-1"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022232Z:3536f45d-afc7-473f-8b37-bad9de7684d7", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc0011d2040>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "mana... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:192]: Unexpected error: <fmt.wrapError \| 0xc0011d2100>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-t8jz2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-t8jz2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00086f3e0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/58e1c38c-4da4-44dd-994d-8cbc32ac06f7", ], "X-Ms-Correlation-Request-Id": [ "3536f45d-afc7-473f-8b37-bad9de7684d7", ], "Date": [ "Thu, 02 Apr 2026 02:22:32 GMT", ], "Content-Type": ["application/json"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: F4CED9AE58F244F885602E16952E2D6D Ref B: CH1AA2020617037 Ref C: 2026-04-02T02:22:31Z", ], "X-Ms-Request-Id": [ "f2afc4a3-ed79-4022-adac-dc064151bd9a", ], "X-Envoy-Upstream-Service-Time": ["744"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Cache": ["CONFIG_NOCACHE"], "Expires": ["-1"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Routing-Request-Id": [ "UKSOUTH:20260402T022232Z:3536f45d-afc7-473f-8b37-bad9de7684d7", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc0011d2040>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "mana... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred