CIHealth Run Log

Environment: DEV

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:46:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5f83abe	n/a	n/a	passed	#4375 (closed)	0	n/a
23:42:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head bc6d993 · merge 0b33790	n/a	n/a	passed	#4376 (merged)	0	n/a
23:41:00	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 489e59b · merge 489e59b	n/a	n/a	passed	#4373 (merged)	0	n/a
22:25:51	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head fee956d	n/a	unknown	failed	#3973 (open)	1	non-artifact-backed failure (no junit artifacts)
21:59:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head d1097e4 · merge d1097e4	n/a	provision	failed	#4374 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 21:59:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-09T22:52:02.814Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-09T22:53:22.684Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=2ec4a404-3ad3-4659-b870-94473c8bdde3 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=406507b1-68b1-406a-a5a6-a4cd20e8bb09 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj0775040.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj0775040.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j0775040 -n arohcp-prow-dbcs-j0775040 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0775040/providers/Microsoft.Network/dnszones/j0775040.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j0775040-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j0775040-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set global.imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j0775040-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"700e186e-1b96-4480-8658-aeda0b3bd4f0\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j0775040-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0775040-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j0775040-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
21:55:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0289e5c · merge 0289e5c	n/a	e2e	failed	#4329 (merged)	1	e2e (1) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 21:55:16 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error: <fmt.wrapError \| 0xc000945460>: failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-w7l2w2" for cluster="ea-cluster": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-w7l2w2\" for cluster=\"ea-cluster\": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0007eb350>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "c61ab021-ab32-4e23-a190-d1d48acb7623", ], "Date": [ "Mon, 09 Mar 2026 23:29:48 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["112"], }, Body: <exported.nopClosingBytesReader \| 0xc000944fe0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j9590400.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-09T23:29:39Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error: <fmt.wrapError \| 0xc000945460>: failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-w7l2w2" for cluster="ea-cluster": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-w7l2w2\" for cluster=\"ea-cluster\": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0007eb350>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "c61ab021-ab32-4e23-a190-d1d48acb7623", ], "Date": [ "Mon, 09 Mar 2026 23:29:48 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["112"], }, Body: <exported.nopClosingBytesReader \| 0xc000944fe0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j9590400.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-09T23:29:39Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
21:40:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 66a5158	n/a	provision	failed	⚠#4272 (open)	1	provision (1) failed post-install: an error on the server ("Internal Server Error: \\"/apis/batch/v1/namespaces/velero/jobs/velero-install\\": Post \\"<url>": EOF") has prevented the request from succeeding (delete jobs.batch velero-install) 21:40:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/management/deploy-velero · suite step graph Show raw failure time=2026-03-09T22:32:44.487Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero description="Step deploy-velero\n Kind: Helm\n" time=2026-03-09T22:32:45.064Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero values="map[azureCloud:AzurePublicCloud azurePlugin:map[digest:sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e repository:oadp/oadp-velero-plugin-for-microsoft-azure-rhel9] bucket:backups clientId:4e5fb508-c971-49fd-b15a-d88ea32903e7 configuration:map[backupPrefix:velero deployNodeAgent:true features:EnableCSI uploaderType:kopia] hypershiftPlugin:map[digest:sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e repository:oadp/oadp-hypershift-velero-plugin-rhel9] imageRegistry:arohcpsvcdev.azurecr.io installJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f installer:map[activeDeadlineSeconds:1800 backoffLimit:3] nodeSelector:map[aro-hcp.azure.com/role:infra] pullBinding:map[workloadIdentityClientId:88657b68-2517-4b35-937a-9968ac0bf3fc workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] resourceGroup:hcp-underlay-prow-j2687744-mgmt-1 storageAccount:oadpprowj26877441 subscriptionId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]] veleroServer:map[digest:sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b repository:oadp/oadp-velero-rhel9]]" time=2026-03-09T22:32:45.064Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero time=2026-03-09T22:32:45.354Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero namespace=velero time=2026-03-09T22:32:45.663Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero dryRun=false time=2026-03-09T22:32:45.663Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero time=2026-03-09T22:32:45.775Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero time=2026-03-09T22:33:28.396Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/velero/jobs/velero-install\\\": Post \\\"https://guard.69af471ce4e3850001c60012.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": EOF\") has prevented the request from succeeding (delete jobs.batch velero-install)" time=2026-03-09T22:33:28.396Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero time=2026-03-09T22:33:28.396Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero time=2026-03-09T22:33:28.548Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero err="failed to get status for release velero: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/velero/secrets?labelSelector=name%3Dvelero%2Cowner%3Dhelm\\\": Post \\\"https://guard.69af471ce4e3850001c60012.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69af471ce4e3850001c60012.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get secrets)" time=2026-03-09T22:33:28.548Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero time=2026-03-09T22:33:28.548Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=deploy-velero err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/velero/jobs/velero-install\\\": Post \\\"https://guard.69af471ce4e3850001c60012.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": EOF\") has prevented the request from succeeding (delete jobs.batch velero-install)"
21:09:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e680d6c · merge 506b932	n/a	provision	failed	★#4372 (merged)	1	provision (1) Unauthorized 21:09:27 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-crds · suite step graph Show raw failure time=2026-03-09T22:03:06.636Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds description="Step deploy-mce-crds\n Kind: Helm\n" time=2026-03-09T22:03:07.168Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds values=map[] time=2026-03-09T22:03:07.168Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-09T22:03:07.658Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds namespace=multicluster-engine time=2026-03-09T22:03:08.145Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds dryRun=false time=2026-03-09T22:03:08.145Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-09T22:03:08.459Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-09T22:03:11.241Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err=Unauthorized time=2026-03-09T22:03:11.241Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-09T22:03:11.241Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-09T22:03:11.469Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds release=mce-crds namespace=multicluster-engine status=failed description="Release \"mce-crds\" failed: Unauthorized" values=map[] time=2026-03-09T22:03:11.469Z level=INFO msg="No resource info available in release, skipping resource diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds infoNil=false resourceCount=0 time=2026-03-09T22:03:11.470Z level=INFO msg="Kusto deep link for namespace-level troubleshooting" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds kustoLinkNamespace="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/3yPzWqFMBSE9z7FISsFlRihtIJv0GVXbkqMg00xPyRHpODDF7uwd3V3w8cMM2OCZ2090ntYc3HS8YUEYuuQWbtIM/gAPJWLZly4FEqql0b2jXz7UGqQ/SBfJ1FR29ITT9dNoqruAq8dctQGn5eicSTh9o2t2fbMSA38aj1EcVJM4RuG/yfVtIW1phiWv2xxUkgLEs0/NyOdTf1wQmfzGwAA//91DCRa6AAAAA==" time=2026-03-09T22:03:11.470Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-09T22:03:11.470Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="error running Helm release deployment Step, failed to deploy helm release: Unauthorized"
20:25:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0289e5c · merge 0289e5c	n/a	provision	failed	★#4329 (merged)	1	provision (1) failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml failed: rolebindings.rbac.authorization.k8s.io "prometheus-admission" is forbidden: user 20:25:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T21:17:09.631Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T21:17:10.317Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:e7827e7a-287a-482f-963b-217e1f5fe28e]] prometheusSpec:map[externalLabels:map[cluster:prow-j5420672-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j5420672-svc-5m0n.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-d185178561f8471d8085adb06d8e0e9f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T21:17:10.317Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T21:17:10.677Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-09T21:17:11.040Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-09T21:17:11.040Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T21:17:11.148Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T21:17:32.089Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml failed: rolebindings.rbac.authorization.k8s.io \"prometheus-admission\" is forbidden: user \"c209f8df-52ae-48fb-98ea-380f58b04652\" (groups=[\"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"secrets\"], Verbs:[\"get\" \"create\"]}" time=2026-03-09T21:17:32.089Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T21:17:32.089Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T21:17:32.242Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to get status for release arohcp-monitor: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/prometheus/secrets?labelSelector=name%3Darohcp-monitor%2Cowner%3Dhelm\\\": Post \\\"https://guard.69af35a704d2a3000103b7b6.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69af35a704d2a3000103b7b6.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get secrets)" time=2026-03-09T21:17:32.243Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T21:17:32.243Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml failed: rolebindings.rbac.authorization.k8s.io \"prometheus-admission\" is forbidden: user \"c209f8df-52ae-48fb-98ea-380f58b04652\" (groups=[\"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"secrets\"], Verbs:[\"get\" \"create\"]}"
20:20:17	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 489e59b · merge 489e59b	n/a	provision	failed	★#4373 (merged)	1	provision (1) create: failed to create: Unauthorized 20:20:17 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/management/deploy · suite step graph Show raw failure time=2026-03-09T21:17:10.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-09T21:17:11.429Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy values="map[azure:map[clientId:7f382cdb-59e9-4015-8c3d-d83c2ecb660a cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-prow-maestro-j1981440.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j1981440-mgmt-1 credsKeyVault:map[name:ah-prow-mg-j1981440-1 secret:hcp-underlay-j1981440-mgmt-1] deployment:map[zoneCount:4] glog_v:2 image:map[digest:sha256:391b002a10ffd994fa4f47e4d1096bac91e512769c0073df782f3cd8d8fe43e7 registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:3 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:2dccfca276d1e7611442dc997ac15055f76c4caff92b668b40907cdd68fffa14 registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]" time=2026-03-09T21:17:11.430Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-09T21:17:11.665Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy namespace=maestro time=2026-03-09T21:17:11.950Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy dryRun=false time=2026-03-09T21:17:11.950Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-09T21:17:12.064Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-09T21:17:17.132Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="create: failed to create: Unauthorized" time=2026-03-09T21:17:17.132Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-09T21:17:17.132Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-09T21:17:18.078Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="failed to get status for release maestro-agent: query: failed to query with labels: Unauthorized" time=2026-03-09T21:17:18.078Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-09T21:17:18.079Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: create: failed to create: Unauthorized"
20:20:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 11ecb87 · merge e0b528c	n/a	unknown	failed	#4371 (merged)	1	non-artifact-backed failure (no junit artifacts)
20:05:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head cdd4611 · merge 489e59b	n/a	unknown	failed	#4373 (merged)	1	non-artifact-backed failure (no junit artifacts)
19:49:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 466bfe5 · merge 506b932	n/a	provision	failed	⚠#4372 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code GetMSICredentialError; detail message Get managed identity credential failed. Details: Category: ClientError; Code: NotFound; SubCode: NotFound; Message: unexpected response from MSI data plane, length of returned certificate: 0; Dependency: MSI Credential; AKSTeam: RP; provider Microsoft.ContainerService 19:49:44 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-03-09T20:23:54.870Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-03-09T20:23:57.463Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-03-09T20:23:59.226Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=9d1c3c7063ba86ccef351ee518270ecb2de5671fe2fe32ffbe110f90b1faa36a time=2026-03-09T20:28:30.393Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0742912-svc/providers/Microsoft.Resources/deployments/9d1c3c7063ba86ccef351ee518270ecb2de5671fe2fe32ffbe110f90b1faa36a/operationStatuses/08584285190472100992\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0742912-svc/providers/Microsoft.Resources/deployments/cluster-kgxhfhv2tbiuw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"BadRequest\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0742912-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0742912-svc\\\",\\r\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"GetMSICredentialError\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Get managed identity credential failed. Details: Category: ClientError; Code: NotFound; SubCode: NotFound; Message: unexpected response from MSI data plane, length of returned certificate: 0; Dependency: MSI Credential; AKSTeam: RP\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
19:34:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head c7c9d81 · merge c7c9d81	n/a	provision	failed	★#4366 (merged)	1	provision (1) failed to install CRD crds/crd-prometheuses.yaml: Unauthorized 19:34:07 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T20:28:06.625Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T20:28:07.464Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:ebccaf49-586b-4097-9337-4377e4b3d590]] prometheusSpec:map[externalLabels:map[cluster:prow-j5980032-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j5980032-svc-mz7g.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-d2bd77058e8c421ca34156ba89ea599d/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T20:28:07.464Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:28:07.893Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-09T20:28:08.224Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-09T20:28:08.224Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:28:08.395Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:28:11.917Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to install CRD crds/crd-prometheuses.yaml: Unauthorized" time=2026-03-09T20:28:11.917Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:28:11.917Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:28:12.034Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to get status for release arohcp-monitor: release: not found" time=2026-03-09T20:28:12.034Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:28:12.035Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to install CRD crds/crd-prometheuses.yaml: Unauthorized"
19:27:43	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 40bc11d · merge 40bc11d	n/a	provision	failed	★#4313 (merged)	1	provision (1) failed post-install: warning: Hook post-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml failed: Unauthorized 19:27:43 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T20:30:06.402Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T20:30:07.054Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:18377044-6808-4a0d-8226-3c720414e5bf]] prometheusSpec:map[externalLabels:map[cluster:prow-j9303424-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9303424-svc-gdad.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-66f1ce5fcdd547c583736426bc362eb0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T20:30:07.054Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:30:07.387Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-09T20:30:07.806Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-09T20:30:07.806Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:30:07.984Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:32:29.208Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed post-install: warning: Hook post-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml failed: Unauthorized" time=2026-03-09T20:32:29.208Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:32:29.208Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:32:30.035Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to get status for release arohcp-monitor: query: failed to query with labels: Unauthorized" time=2026-03-09T20:32:30.035Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:32:30.035Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: warning: Hook post-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml failed: Unauthorized"
19:10:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 523a271 · merge 523a271	n/a	provision	failed	#4363 (merged)	1	provision (1) Unauthorized Unauthorized Unauthorized 19:10:49 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/deploy · suite step graph Show raw failure time=2026-03-09T20:12:11.832Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-09T20:12:12.478Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-09T20:12:12.478Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T20:12:12.748Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy namespace=openshift-route-monitor-operator time=2026-03-09T20:12:13.031Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy dryRun=false time=2026-03-09T20:12:13.031Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T20:12:13.138Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T20:12:23.616Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="Unauthorized\nUnauthorized\nUnauthorized" time=2026-03-09T20:12:23.616Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T20:12:23.616Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T20:12:26.076Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy release=route-monitor-operator namespace=openshift-route-monitor-operator status=failed description="Release \"route-monitor-operator\" failed: Unauthorized\nUnauthorized\nUnauthorized" values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-09T20:12:26.077Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yTT2/TQBTE7/4UTz4lkl3Zbv44QT0A4oRAiHICcVjvTpzN/nN310mD8uGRCaFRKaitep0d/d57o51EI5JHcL3nCHRFgkUWWaMxcs0GPL6XVixD9NK2GR2lj8zgj4Qt7C8ldIyf5PG3hIgo/aQZh4GNaUapYiE3zsroBku+ZlwNsutgw1quYu5dH3Gy5K6DZ9H5NLvHeiOtkLZ9CaTTku9fAHQNv5Ucrzl3/fHYh/152IcI8xT0W92HCP/ZafyDmy9qvdveuk6VfGL4/vZmUodWL6pmNm+bds93hVJi4kXw0wHxAPks0r8H5CU38x/cV2XNq5vVpi8bsdpA+LZeTKcNpFIFt7N5p2tWP2fAsy84hdJ5ZxDX6EOu6vCUdD8wy1qI35te4/xzCaxYr+OjcMn3V4nqG3iLiPBu6ERIDrRbw4OiNAiRmY4axB1gaSRYxCCP0qqoZnlxmReLL1WxLKtlefk1HdPFBf3HUw2ecXKgjZOWlLTiSloLf9ZkZ+muwOfNvV/Z5ECdd8Pr3aIZdU4czY+DZKRdm5FBCKwdkM4LeGr2Z8cLBP4zAAD//8VOmdJvBAAA" time=2026-03-09T20:12:26.077Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T20:12:26.078Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Unauthorized\nUnauthorized\nUnauthorized"
19:07:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 523a271 · merge 523a271	n/a	provision	failed	#4363 (merged)	1	provision (1) failed to apply namespace prometheus: Unauthorized 19:07:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T20:02:27.496Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T20:02:28.124Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:e24f4666-9dad-4bcf-a6a8-810afacceda9]] prometheusSpec:map[externalLabels:map[cluster:prow-j6226688-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6226688-svc-itz1.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-2883643e837c4ecea61b4287c67a15e9/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T20:02:28.124Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T20:02:29.126Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace prometheus: Unauthorized"
19:05:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head a72618f · merge a72618f	n/a	unknown	failed	#4362 (merged)	1	non-artifact-backed failure (no junit artifacts)
18:29:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 11ecb87 · merge e0b528c	n/a	unknown	failed	#4371 (merged)	1	non-artifact-backed failure (no junit artifacts)
17:55:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 5b9af08 · merge 5b9af08	n/a	provision	failed	★#4361 (merged)	1	provision (1) resource not ready, name: sessiongate, kind: Deployment, status: Unknown context deadline exceeded 17:55:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/service/deploy · suite step graph Show raw failure time=2026-03-09T19:05:18.347Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-09T19:05:19.460Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:0edffd03-5e66-4526-b4d5-f391fcc114f1 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j7874176] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j7874176.hcpsvc.osadev.cloud] image:map[digest:sha256:c1ae07dbef1925a1b2f02410f442fdc94d2a2ac8c3ff32d64db02d07be4d69fb registry:arohcpsvcdev.azurecr.io repository:ci-op-9fxy9dxs/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-9fxy9dxs/pipeline:pull workloadIdentityClientId:c2d7b981-86f9-402f-b460-b243cd4e9e83 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:09b8b89e-472d-4280-a194-30a949204bef workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-09T19:05:19.460Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-09T19:05:19.691Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate time=2026-03-09T19:05:19.993Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false time=2026-03-09T19:05:19.993Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-09T19:05:20.106Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-09T19:10:28.975Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="resource not ready, name: sessiongate, kind: Deployment, status: Unknown\ncontext deadline exceeded" time=2026-03-09T19:10:28.975Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-09T19:10:28.975Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-09T19:10:33.464Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: resource not ready, name: sessiongate, kind: Deployment, status: Unknown\ncontext deadline exceeded" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:0edffd03-5e66-4526-b4d5-f391fcc114f1 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j7874176] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j7874176.hcpsvc.osadev.cloud] image:map[digest:sha256:c1ae07dbef1925a1b2f02410f442fdc94d2a2ac8c3ff32d64db02d07be4d69fb registry:arohcpsvcdev.azurecr.io repository:ci-op-9fxy9dxs/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-9fxy9dxs/pipeline:pull workloadIdentityClientId:c2d7b981-86f9-402f-b460-b243cd4e9e83 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:09b8b89e-472d-4280-a194-30a949204bef workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-09T19:10:33.466Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5RUTW/bMAy9+1cIPjVAZKQdNqwZeujHgAEDhiDrqcMOsszaamXRI6kWKfLjBzfJYjjOx24S/fT46Ecy8SCKgDGSBVZXqjBixOQezjB/AivfXSimLORCOVar0A9Tw78QvEB4j3Bj7CY8+pUopVR6B43HRQ1B0rFKGZgdhtII7FyP4LUFEvfobHsmeCTgCqgFmWfWjsWhdqEkYN5wXVuaRe9vXChcKFtoE73X+fY+IOA6SoXk3ow4DDP0zi7ek3iPr/00/09Qg5CzrPn4nzjIE1Cq/TV8u7+fzTEOUA/CbyML1vN1C9zBowuuTdqBc9Z5lxlCXdkmM2+RILNYt8gN2xx9P5G2GITQ+5VhAxIOP9IeTAGkwYPtCeuT3Aya2+U6bP9pHD09Rzh/Ar04e6IZc/gTgaV1H4K07b5bsKYVSJso1d6klkBmhC+uALr1ZqdhNdvm4PysdV9bi/HU+b31kQVoyM/1WdcmmBI2I54mv78kzzEHCiDAX9tlwslSvVZAoMTVwGLqRuUgrwBBnRVGoA2fpReTi0968kFPLu/PL6eTj9Pzy4d0pLJM7cOcT6YXnx/S0ShZqid0QT27UFy5EIA6KxCD2m6+7srr77pkqRrC9utW6Fg1WKzAp5GMlcdyrGpgNmVLiVQAqXzRKb4Atn8DAAD//6g2/yKoBQAA" time=2026-03-09T19:10:33.469Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-09T19:10:33.469Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: sessiongate, kind: Deployment, status: Unknown\ncontext deadline exceeded"
17:44:02	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 5b9af08 · merge 5b9af08	n/a	provision	failed	#4361 (merged)	1	provision (1) failed post-install: Unauthorized 17:44:02 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T18:59:06.251Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T18:59:06.981Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:5a237c08-583c-4b4b-8951-44ba1d90f3e8]] prometheusSpec:map[externalLabels:map[cluster:prow-j7720448-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j7720448-svc-kl72.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-78fa24611df44506bfdbdacb3d78fb5f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T18:59:06.981Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T18:59:07.398Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-09T18:59:07.671Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-09T18:59:07.671Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T18:59:07.825Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:01:05.632Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed post-install: Unauthorized" time=2026-03-09T19:01:05.632Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:01:05.632Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:01:25.273Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed post-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:5a237c08-583c-4b4b-8951-44ba1d90f3e8]] prometheusSpec:map[externalLabels:map[cluster:prow-j7720448-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j7720448-svc-kl72.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-78fa24611df44506bfdbdacb3d78fb5f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T19:01:25.274Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yTT2/TQBDF7/4UI58Sya7cogIx6gEKJ1QOgEAq4rDeHZxtvDurmXGrSPnwyHFDDKUoIj36+c1v/mhf1qECo1DPFgUuwBk1apoOZ9TcoNX3PrpalH1sCxilDybgLwlvMW4VScbu5Pm3DAAgv6T4w7dXJuUF5CaYMqCyt1IKqvrYSmm3jjA6Vn2DpaxFMeTFSPhiOu/M4P2KzZJoNSJ7NuopDkWJKaAusZfSuOBF7vUd4bLrRZE/UofbKZiWNpWBolficmypRnE32z9qJ60oIRslfsT9xkfnY3tMwwniv/rui6beT8i33h5+iSllJLzF1NE6YNQngzyy38Oyq16Pegv327+2lvqj5n8I+v3ck6/s+6tswHJERXk3xEWyDdwtkRHUBxQ1IUGDeocYYeaM4iDP8rPq7HlZPSurxefTl/X5oq5eXOdzODmBv3sWdXVaV+fX+XyebeCGfISVj+7Cx4g8CTlF2Gd7Guo/05xtIDENf/eDFpDIjebDIAV01BYQUMS0A5LYIUOznizvUOzPAAAA//8qdpMtigQAAA==" time=2026-03-09T19:01:25.276Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:01:25.276Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
17:35:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e430bc4 · merge a36f37a	n/a	provision	failed	#4205 (merged)	1	provision (1) failed pre-install: Unauthorized 17:35:54 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T19:14:40.550Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T19:14:41.306Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:90cdf569-1710-4521-9fc6-1f3ff3ea17c5]] prometheusSpec:map[externalLabels:map[cluster:prow-j5790592-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j5790592-svc-pikb.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8c209c0d1f4c49e4b5c4c592e6250129/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T19:14:41.306Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:14:41.619Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-09T19:14:42.004Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-09T19:14:42.004Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:14:42.109Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:16:11.008Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: Unauthorized" time=2026-03-09T19:16:11.008Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:16:11.009Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:16:38.898Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed pre-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:90cdf569-1710-4521-9fc6-1f3ff3ea17c5]] prometheusSpec:map[externalLabels:map[cluster:prow-j5790592-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j5790592-svc-pikb.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8c209c0d1f4c49e4b5c4c592e6250129/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T19:16:38.898Z level=INFO msg="No resource info available in release, skipping resource diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus infoNil=false resourceCount=0 time=2026-03-09T19:16:38.899Z level=INFO msg="Kusto deep link for namespace-level troubleshooting" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus kustoLinkNamespace="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/3SOPYqEQBSEc0/x6EhBxXZFUPAGG25ksrRtoS7bP3S/QQY8/OAEziSTFR9V1KedZbVZhG+3xOSgfUUA8WYQWRlPE3gHLKWzYpw4FXVVt0X1VVTdj+x62fRNPYqMypI+dtpeylFk2XVglUH0SuP3TDQMJHxwBrziFkVykA/uD5pfJjn9uyUn7+bnJDnIhRmBpvvFSEWdv7mrqB8BAAD//xo9pYDfAAAA" time=2026-03-09T19:16:38.899Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T19:16:38.899Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed pre-install: Unauthorized"
17:08:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0f2aca0 · merge 1892906	n/a	provision	failed	#3820 (merged)	1	provision (1) failed post-install: Unauthorized 17:08:50 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-09T17:59:19.656Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T17:59:20.363Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:d01706ad-9272-4795-9df0-cd12fe407df6]] prometheusSpec:map[externalLabels:map[cluster:prow-j6606592-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j6606592-mgmt-1-x3ef.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-f37991ed6afb4738b97c5931ddf103be/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6606592-mgmt-1-x3ef.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-86a5f5e8e8dc4484a806e3a406e53c23/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T17:59:20.363Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T17:59:20.636Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-09T17:59:20.958Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-09T17:59:20.958Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T17:59:21.111Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T18:00:10.302Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: Unauthorized" time=2026-03-09T18:00:10.302Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T18:00:10.302Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T18:00:11.064Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed to get status for release arohcp-monitor: query: failed to query with labels: Unauthorized" time=2026-03-09T18:00:11.064Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T18:00:11.065Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
16:57:58	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ac18f09	n/a	provision	failed	#3973 (open)	1	provision (1) failed pre-install: Unauthorized 16:57:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-09T17:45:11.053Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T17:45:11.665Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:9df25c9b-5dca-4c19-badc-fc546b07c864]] prometheusSpec:map[externalLabels:map[cluster:prow-j6492928-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6492928-svc-w1w1.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-c0f374669a024c3b8646e4b2f2fdcccc/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T17:45:11.665Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T17:45:12.058Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-09T17:45:12.443Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-09T17:45:12.443Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T17:45:12.553Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T17:46:26.498Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: Unauthorized" time=2026-03-09T17:46:26.498Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T17:46:26.498Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T17:46:49.466Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed pre-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:9df25c9b-5dca-4c19-badc-fc546b07c864]] prometheusSpec:map[externalLabels:map[cluster:prow-j6492928-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6492928-svc-w1w1.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-c0f374669a024c3b8646e4b2f2fdcccc/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T17:46:49.467Z level=INFO msg="No resource info available in release, skipping resource diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus infoNil=false resourceCount=0 time=2026-03-09T17:46:49.467Z level=INFO msg="Kusto deep link for namespace-level troubleshooting" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus kustoLinkNamespace="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/3SOv4qDQBjEe5/iYysFFd278zjBN7gylU1Y10EN2T/sfkECPnwwhUmTbvgxw/y0s6wWi/DvpphstM4IIF4MIivjaQCvgKV0VIwdp0JWsimqr6L6O9W/7fdPW8teZFSW9LHTtLLpRZYdB1YZRK80znuiriPhgzPgGbcoko18cBdofpnkdHVTTt6Nz0mykQsjAg33g5GKOn9zV1E/AgAA//841rbS3wAAAA==" time=2026-03-09T17:46:49.467Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-09T17:46:49.468Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed pre-install: Unauthorized"
16:52:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a72618f · merge a72618f	n/a	unknown	failed	#4362 (merged)	1	non-artifact-backed failure (no junit artifacts)
16:38:18	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0289e5c · merge 0289e5c	n/a	provision	failed	#4329 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 16:38:18 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-09T17:29:26.530Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-09T17:29:50.041Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=f26802a5-17ea-403a-9a5d-adc7ad08cd71 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=ef2ddc9d-ccea-42c0-919d-e108e1d67595 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj8350848.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj8350848.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j8350848 -n arohcp-prow-dbcs-j8350848 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8350848/providers/Microsoft.Network/dnszones/j8350848.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j8350848-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j8350848-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set global.imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j8350848-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"af9d0d79-b9c1-4f77-8027-fc7adf9326e5\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j8350848-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8350848-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j8350848-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unable to continue with install: could not get information about the resource AuthorizationPolicy \"allow-metrics\" in namespace \"clusters-service\": Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
16:35:26	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 0289e5c · merge 0289e5c	n/a	provision	failed	#4329 (merged)	1	provision (1) failed post-install: Unauthorized 16:35:26 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce · suite step graph Show raw failure time=2026-03-09T17:34:17.359Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce description="Step deploy-mce\n Kind: Helm\n" time=2026-03-09T17:34:17.954Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-09T17:34:17.954Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-09T17:34:18.012Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce namespace=multicluster-engine time=2026-03-09T17:34:18.121Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce dryRun=false time=2026-03-09T17:34:18.121Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-09T17:34:18.230Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-09T17:35:35.855Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed post-install: Unauthorized" time=2026-03-09T17:35:35.855Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-09T17:35:35.855Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-09T17:35:39.803Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce release=mce namespace=multicluster-engine status=pending-install description="Initial install underway" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-09T17:35:39.804Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yTQW+bQBCF7/yKEadYgsjGUBxXObRVTlV6aHpK1cOyPMMG2CWzu7aI/OMraidxq1L3kOubt9880LyghSOGNZ4lLF1TKZxwomhxYYoHSPdZ6XJtHStdRXSQvogOLxK20L8U2wv5LM++B0RE4R0kw4URhZ1vnZKttw4cQ1dKIzY9WDjD8Q5FbUwz4QujA+zTQf1qWkwRe1936TLfZr25SgeT9rovqzzLUqGSnNunLu1WPnus86IcEX8hf1S6VLp64wX/ypykQ+/zZjM4O3CySLVSpmZ436wSNN1ys3vi2rQWj1OI0y1n8r/dsttxdPxxN8fJ7w/woj6/uQNvlcSt0coZHidHa6s2kINsEVsnHOIOjpW08TY5k+JI/CCl8frsoU3Bgh/vg8YXYA0HezNetA32tKvBIKc6WCe6ngq4HaDpohQOo3wRJvPkXTxfxvOrb4t8vUzXi9V9OKPLS5r0ZOtldh/OZsGeHozS1ChdXiutwSc9NJpe63fauz8LF+ypZzNOX4NG1JvyYP4/SEStqSLqYK2oRqThEkzFcPLxJaz8GQAA//8NlsxjLQQAAA==" time=2026-03-09T17:35:39.805Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-09T17:35:39.805Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
16:35:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2a2dae4	n/a	unknown	failed	#4370 (open)	1	non-artifact-backed failure (no junit artifacts)
16:22:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ecf4a8c	n/a	provision	failed	⚠#4369 (open)	1	provision (1) "mce-config" has no deployed releases 16:22:36 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-09T17:14:49.228Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-09T17:14:50.219Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-09T17:14:50.219Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T17:14:50.275Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-09T17:14:50.385Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-09T17:14:50.385Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T17:14:51.142Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config numVersions=0 time=2026-03-09T17:14:51.246Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="\"mce-config\" has no deployed releases" time=2026-03-09T17:14:51.246Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T17:14:51.246Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T17:14:51.350Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: release: not found" time=2026-03-09T17:14:51.350Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T17:14:51.350Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: \"mce-config\" has no deployed releases"
16:18:18	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e7f1e37 · merge e7f1e37	n/a	unknown	failed	#4244 (merged)	1	non-artifact-backed failure (no junit artifacts)
16:13:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 5cda440 · merge c117b0a	n/a	provision	failed	★#4350 (merged)	1	provision (1) Unauthorized 16:13:49 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/deploy · suite step graph Show raw failure time=2026-03-09T17:12:43.063Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-09T17:12:43.686Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-09T17:12:43.686Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T17:12:43.943Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy namespace=openshift-route-monitor-operator time=2026-03-09T17:12:44.217Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy dryRun=false time=2026-03-09T17:12:44.217Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T17:12:44.322Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T17:12:53.834Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err=Unauthorized time=2026-03-09T17:12:53.834Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T17:12:53.834Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T17:12:54.562Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="failed to get status for release route-monitor-operator: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-09T17:12:54.562Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-09T17:12:54.562Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Unauthorized"
15:32:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a72618f · merge a72618f	n/a	provision	failed	★#4362 (merged)	1	provision (1) failed post-install: an error on the server ("Internal Server Error: \\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\": Post \\"<url>": proxy error from localhost:9443 while dialing 15:32:14 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-09T16:23:31.884Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-09T16:23:32.553Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-09T16:23:32.553Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:23:32.610Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-09T16:23:32.721Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-09T16:23:32.721Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:23:32.834Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:25:18.277Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aef03a53dc340001ad4fd8.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69aef03a53dc340001ad4fd8.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" time=2026-03-09T16:25:18.277Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:25:18.277Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:25:18.408Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine/secrets?labelSelector=name%3Dmce-config%2Cowner%3Dhelm\\\": Post \\\"https://guard.69aef03a53dc340001ad4fd8.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69aef03a53dc340001ad4fd8.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get secrets)" time=2026-03-09T16:25:18.408Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:25:18.409Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aef03a53dc340001ad4fd8.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69aef03a53dc340001ad4fd8.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)"
15:29:19	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head a72618f · merge a72618f	n/a	provision	failed	★#4362 (merged)	1	provision (1) tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\ 15:29:19 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-09T16:22:01.448Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-09T16:22:02.113Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-09T16:22:02.113Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:22:02.170Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-09T16:22:02.284Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-09T16:22:02.284Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:22:02.397Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:23:28.267Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aef02a3078b40001365376.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" time=2026-03-09T16:23:28.267Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:23:28.267Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:23:28.415Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine/secrets?labelSelector=name%3Dmce-config%2Cowner%3Dhelm\\\": Post \\\"https://guard.69aef02a3078b40001365376.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (get secrets)" time=2026-03-09T16:23:28.415Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T16:23:28.415Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aef02a3078b40001365376.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)"
15:25:58	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 78f5869 · merge 78f5869	n/a	n/a	passed	#3838 (merged)	0	n/a
15:18:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 361dd4b	n/a	provision	failed	#4157 (open)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 15:18:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-09T16:11:09.882Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-09T16:14:13.473Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=b54f15bd-5909-4f0f-a1b1-b22df91e2f9a && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=3b6966a1-b1aa-421e-8a73-5f22939748ba && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj5477504.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj5477504.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j5477504 -n arohcp-prow-dbcs-j5477504 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5477504/providers/Microsoft.Network/dnszones/j5477504.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j5477504-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j5477504-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set global.imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j5477504-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"5e55a69f-a797-48e4-840e-895adda3ba18\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j5477504-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5477504-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j5477504-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/clusters-service/persistentvolumeclaims/ocm-cs-db\\\": Post \\\"https://guard.69aeed4dab496b000129e332.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (get persistentvolumeclaims ocm-cs-db)\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
14:33:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e7f1e37 · merge e7f1e37	n/a	provision	failed	★#4244 (merged)	1	provision (1) tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\ 14:33:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-09T15:22:27.370Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-09T15:22:28.065Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-09T15:22:28.065Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T15:22:28.126Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-09T15:22:28.246Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-09T15:22:28.246Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T15:22:28.363Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T15:24:21.061Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aee1feab496b000129e323.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" time=2026-03-09T15:24:21.061Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T15:24:21.061Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T15:24:25.786Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Release \"mce-config\" failed: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aee1feab496b000129e323.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-09T15:24:25.788Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6SUz27bMAzG734KwqcGiIrU+4PNQw9dt1OxDdh26rCDLLGqGpk0JDlBgDz8oNiOvaxtgu7ggz+RP5GfCGYOI3gM3HqFAS5ByyijrByecfWAKt5Y0mWI3pKZQyd9lTXuJVwh7ZTQSDXIs18ZAEB+49oQ0TuMV1ozXTPdWZPPIXespBOqO35E6NK/s8N0aLwS6fP9f926aPtYgWQs4ZBy3alfJEmDNVK6+BulJLW7XDTsrNoIxRQ9O9fdfjzb8Ao9SVI4EO68rHHNfjkFjA2PvRrHlXTTqB/oV1bhlVLcUkwny32ekMmpSX1iVRzp+V9cMqsvs8MFeZpvg+PcIA2vIeq9HaXxquzl4TFeQOhK6xs9SvtoSVsyz0GP2/cy7kG71ZgxHdEJaJjUv8w/SH3mFZ6gnQ74hI3jTar9v6dqN/ojb5znDnXPIaIWai8/hsp+f8iWbYWeMGL4nFZFyLawvkePEG2NIcq6gQrjGpHgTMuIST7Li0XxVixeicX7nxdvyqIoi3e3+QzOz+HJmNdlcXGbz2bZFh7YEiwt6UtLhH6y4Jhg3GvThXa4ybItNJ7T6VjoHBrWXfBpkDk4NnOoMQRpEpK9Rg/VZtK8xqD+BAAA///eIDIahgUAAA==" time=2026-03-09T15:24:25.790Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-09T15:24:25.790Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69aee1feab496b000129e323.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)"
14:27:19	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head a72618f · merge a72618f	n/a	provision	failed	★#4362 (merged)	1	provision (1) unable to continue with install: could not get information about the resource PodMonitor "prometheus" in namespace "<namespace>": Unauthorized 14:27:19 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-09T15:12:33.138Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-09T15:12:33.779Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:7c74a715af2c94cb734785b4d3ea1357b4f02b88e1e123c622a9cb68b62f669c]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:d19a5f17-9803-4590-8210-047e474e6bef]] prometheusSpec:map[externalLabels:map[cluster:prow-j1775104-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j1775104-mgmt-1-h7u6.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-755a9b7fa00f4d08a8ad438130445a22/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j1775104-mgmt-1-h7u6.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-e9ed56d6e7704d93a2bd5047f0a57cec/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-09T15:12:33.780Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T15:12:34.081Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-09T15:12:34.497Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-09T15:12:34.497Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T15:12:34.672Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T15:12:44.243Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="unable to continue with install: could not get information about the resource PodMonitor \"prometheus\" in namespace \"prometheus\": Unauthorized" time=2026-03-09T15:12:44.243Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T15:12:44.243Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T15:12:44.362Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed to get status for release arohcp-monitor: release: not found" time=2026-03-09T15:12:44.362Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-09T15:12:44.362Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource PodMonitor \"prometheus\" in namespace \"prometheus\": Unauthorized"
13:53:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 120f259 · merge 120f259	n/a	n/a	passed	#4368 (merged)	0	n/a
13:43:18	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0289e5c · merge 0289e5c	n/a	n/a	passed	#4329 (merged)	0	n/a
13:41:34	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 8e77d7a · merge 120f259	n/a	unknown	failed	#4368 (merged)	1	non-artifact-backed failure (no junit artifacts)
13:40:48	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 5b9af08 · merge 5b9af08	n/a	n/a	passed	#4361 (merged)	0	n/a
13:20:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 78f5869 · merge 78f5869	n/a	n/a	passed	#3838 (merged)	0	n/a
13:13:37	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head c7c9d81 · merge c7c9d81	n/a	n/a	passed	#4366 (merged)	0	n/a
13:06:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head e2ff225	n/a	n/a	passed	#4272 (open)	0	n/a
12:25:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 331cb48 · merge 6106740	n/a	n/a	passed	#4367 (merged)	0	n/a
12:22:05	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 523a271 · merge 523a271	n/a	n/a	passed	#4363 (merged)	0	n/a
11:52:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head c7c9d81 · merge c7c9d81	n/a	provision	failed	#4366 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 11:52:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-03-09T12:23:14.260Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-09T12:23:16.741Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-03-09T12:23:18.093Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=4377641c4adac4a3d8c4ed688b61f4475e5a685ffa0600e2ffc07cfa4a0f2cc9 time=2026-03-09T12:24:48.493Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8970368/providers/Microsoft.Resources/deployments/4377641c4adac4a3d8c4ed688b61f4475e5a685ffa0600e2ffc07cfa4a0f2cc9/operationStatuses/08584285478881872548\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8970368/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8970368/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j8970368\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report b937a25f-6d64-43a4-891a-cb61c3c6e1d2:3/9/2026 12:24:38 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:44:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9e94925	n/a	e2e	failed	#4040 (open)	2	e2e (2) ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift 11:44:52 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapError \| 0xc000a4ab20>: failed to start credential request: POST https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: Conflict -------------------------------------------------------------------------------- { "error": { "code": "Conflict", "message": "Cannot request credential while resource is \"provisioning\"", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster" } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00131d920>{ ErrorCode: "Conflict", StatusCode: 409, RawResponse: { Status: "409 Conflict", StatusCode: 409, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["316"], "X-Envoy-Upstream-Service-Time": ["1031"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": ["Conflict"], "X-Ms-Request-Id": [ "237cd894-58fe-4457-8f45-5cffa02fb959", ], "Date": [ "Mon, 09 Mar 2026 12:41:48 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a4aa40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 98, 98, 52, 118, 50, 100, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 316, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j4135552.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredenti... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapError \| 0xc000a4ab20>: failed to start credential request: POST https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: Conflict -------------------------------------------------------------------------------- { "error": { "code": "Conflict", "message": "Cannot request credential while resource is \"provisioning\"", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster" } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00131d920>{ ErrorCode: "Conflict", StatusCode: 409, RawResponse: { Status: "409 Conflict", StatusCode: 409, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["316"], "X-Envoy-Upstream-Service-Time": ["1031"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": ["Conflict"], "X-Ms-Request-Id": [ "237cd894-58fe-4457-8f45-5cffa02fb959", ], "Date": [ "Mon, 09 Mar 2026 12:41:48 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a4aa40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 98, 98, 52, 118, 50, 100, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 316, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j4135552.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bb4v2d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredenti... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift 11:44:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error: <fmt.wrapError \| 0xc0014489e0>: failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-xznwhl": PUT https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1 -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: Conflict -------------------------------------------------------------------------------- { "error": { "code": "Conflict", "message": "Cannot create resource while parent resource is provisioning", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1" } } -------------------------------------------------------------------------------- { msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xznwhl\": PUT https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0014489c0>{ msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xznwhl\": PUT https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ef7ad0>{ ErrorCode: "Conflict", StatusCode: 409, RawResponse: { Status: "409 Conflict", StatusCode: 409, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": ["Conflict"], "X-Ms-Request-Id": [ "6d879c31-98e6-4e93-9efe-1acea3373990", ], "Date": [ "Mon, 09 Mar 2026 12:41:50 GMT", ], "Content-Length": ["345"], "X-Envoy-Upstream-Service-Time": ["2084"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc001448840>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error: <fmt.wrapError \| 0xc0014489e0>: failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-xznwhl": PUT https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1 -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: Conflict -------------------------------------------------------------------------------- { "error": { "code": "Conflict", "message": "Cannot create resource while parent resource is provisioning", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1" } } -------------------------------------------------------------------------------- { msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xznwhl\": PUT https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0014489c0>{ msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xznwhl\": PUT https://rp.j4135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-xznwhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ef7ad0>{ ErrorCode: "Conflict", StatusCode: 409, RawResponse: { Status: "409 Conflict", StatusCode: 409, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": ["Conflict"], "X-Ms-Request-Id": [ "6d879c31-98e6-4e93-9efe-1acea3373990", ], "Date": [ "Mon, 09 Mar 2026 12:41:50 GMT", ], "Content-Length": ["345"], "X-Envoy-Upstream-Service-Time": ["2084"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc001448840>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
10:47:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head eac14df · merge 131806c	n/a	e2e	failed	#4351 (merged)	1	e2e (1) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:47:47 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error: <errors.joinError \| 0xc000f746f0>: cluster listing in non-existent resource group: expected error but none occurred node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <errors.errorString \| 0xc0014f2440>{ s: "cluster listing in non-existent resource group: expected error but none occurred", }, <errors.errorString \| 0xc0014f2c00>{ s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, <errors.errorString \| 0xc0012b70a0>{ s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error: <errors.joinError \| 0xc000f746f0>: cluster listing in non-existent resource group: expected error but none occurred node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <errors.errorString \| 0xc0014f2440>{ s: "cluster listing in non-existent resource group: expected error but none occurred", }, <errors.errorString \| 0xc0014f2c00>{ s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, <errors.errorString \| 0xc0012b70a0>{ s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, ], } occurred
10:35:34	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3832c3f · merge 3832c3f	n/a	n/a	passed	#4317 (merged)	0	n/a
10:32:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 3832c3f · merge 3832c3f	n/a	n/a	passed	#4317 (merged)	0	n/a
10:30:57	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 78f5869 · merge 78f5869	n/a	n/a	passed	#3838 (merged)	0	n/a
09:44:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head f758798 · merge 523a271	n/a	e2e	failed	⚠#4363 (merged)	20	e2e (20) Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure pp-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:38 STEP: creating a resource group @ 03/09/26 10:10:47.829 "ts"="2026-03-09 10:10:47.829823" "level"=0 "msg"="creating resource group" "resourceGroup"="basic-cluster-5dwqb4" STEP: creating cluster parameters @ 03/09/26 10:10:51.891 STEP: creating customer resources @ 03/09/26 10:10:51.891 STEP: creating the HCP cluster @ 03/09/26 10:11:51.116 "ts"="2026-03-09 10:11:51.116841" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster" "resourceGroup"="basic-cluster-5dwqb4" STEP: getting credentials @ 03/09/26 10:23:26.705 STEP: ensuring the cluster is viable @ 03/09/26 10:24:18.259 STEP: creating the node pool @ 03/09/26 10:24:18.625 STEP: verifying nodepool DiskStorageAccountType matches framework default @ 03/09/26 10:32:32.065 STEP: verifying a simple web app can run @ 03/09/26 10:32:32.16 I0309 10:32:32.617568 761 serving_app.go:120] "created resources" namespace="e2e-serving-app-7m5kv" deployment="agnhost-server" service="agnhost" route="agnhost" I0309 10:34:44.085566 761 serving_app.go:198] Get "https://agnhost-e2e-serving-app-7m5kv.apps.aro.r9g5s1z6g5g4g7b.c4ed.j2548992.hcp.osadev.cloud": dial tcp 4.242.208.5:443: connect: connection timed outfailed to get response from routeurlhttps://agnhost-e2e-serving-app-7m5kv.apps.aro.r9g5s1z6g5g4g7b.c4ed.j2548992.hcp.osadev.cloud Route became available in less than 5 minutes: url=https://agnhost-e2e-serving-app-7m5kv.apps.aro.r9g5s1z6g5g4g7b.c4ed.j2548992.hcp.osadev.cloud elapsed=2m11.623483652s got successful response from route: response=HTTP/1.1 200 OK Content-Length: 31 Content-Type: text/plain; charset=utf-8 Date: Mon, 09 Mar 2026 10:34:44 GMT Set-Cookie: 7b24a1275d9da4fa66633acf82aa53e5=fc49160d68b8f372ae3a8c2fd6ec39e3; path=/; HttpOnly; Secure; SameSite=None agnhost-server-8678c6cb89-xx8jm "ts"="2026-03-09 10:34:44.295103" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:34:44.295274" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-zsbbgv" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:34:45.369263" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-zsbbgv" "resourceGroup"="" "ts"="2026-03-09 10:34:46.300877" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz" "ts"="2026-03-09 10:34:46.300998" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz" "ts"="2026-03-09 10:34:46.301090" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:34:46.302428" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-116" "ts"="2026-03-09 10:34:46.302462" "level"=0 "msg"="collecting deployments" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:34:47.018090" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-116" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:34:47.540716" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-5dwqb4" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-03-09 10:34:48.067906" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-116" "deployment"="msi-deployment-116" "ts"="2026-03-09 10:34:48.560596" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-5dwqb4" "deployment"="customer-infra-basic-hcp-cluster-zsbbgv" "ts"="2026-03-09 10:34:49.634787" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:34:49.635001" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:34:49.635157" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:43:34.015898" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:43:34.015954" "level"=0 "msg"="deleting resource group" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:45:03.662128" "level"=0 "msg"="finished deleting created resources" • [2055.843 seconds] ------------------------------ Ran 1 of 1 Specs in 2055.844 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') pp-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:38 STEP: creating a resource group @ 03/09/26 10:10:47.829 "ts"="2026-03-09 10:10:47.829823" "level"=0 "msg"="creating resource group" "resourceGroup"="basic-cluster-5dwqb4" STEP: creating cluster parameters @ 03/09/26 10:10:51.891 STEP: creating customer resources @ 03/09/26 10:10:51.891 STEP: creating the HCP cluster @ 03/09/26 10:11:51.116 "ts"="2026-03-09 10:11:51.116841" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster" "resourceGroup"="basic-cluster-5dwqb4" STEP: getting credentials @ 03/09/26 10:23:26.705 STEP: ensuring the cluster is viable @ 03/09/26 10:24:18.259 STEP: creating the node pool @ 03/09/26 10:24:18.625 STEP: verifying nodepool DiskStorageAccountType matches framework default @ 03/09/26 10:32:32.065 STEP: verifying a simple web app can run @ 03/09/26 10:32:32.16 I0309 10:32:32.617568 761 serving_app.go:120] "created resources" namespace="e2e-serving-app-7m5kv" deployment="agnhost-server" service="agnhost" route="agnhost" I0309 10:34:44.085566 761 serving_app.go:198] Get "https://agnhost-e2e-serving-app-7m5kv.apps.aro.r9g5s1z6g5g4g7b.c4ed.j2548992.hcp.osadev.cloud": dial tcp 4.242.208.5:443: connect: connection timed outfailed to get response from routeurlhttps://agnhost-e2e-serving-app-7m5kv.apps.aro.r9g5s1z6g5g4g7b.c4ed.j2548992.hcp.osadev.cloud Route became available in less than 5 minutes: url=https://agnhost-e2e-serving-app-7m5kv.apps.aro.r9g5s1z6g5g4g7b.c4ed.j2548992.hcp.osadev.cloud elapsed=2m11.623483652s got successful response from route: response=HTTP/1.1 200 OK Content-Length: 31 Content-Type: text/plain; charset=utf-8 Date: Mon, 09 Mar 2026 10:34:44 GMT Set-Cookie: 7b24a1275d9da4fa66633acf82aa53e5=fc49160d68b8f372ae3a8c2fd6ec39e3; path=/; HttpOnly; Secure; SameSite=None agnhost-server-8678c6cb89-xx8jm "ts"="2026-03-09 10:34:44.295103" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:34:44.295274" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-zsbbgv" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:34:45.369263" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-zsbbgv" "resourceGroup"="" "ts"="2026-03-09 10:34:46.300877" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz" "ts"="2026-03-09 10:34:46.300998" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz" "ts"="2026-03-09 10:34:46.301090" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:34:46.302428" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-116" "ts"="2026-03-09 10:34:46.302462" "level"=0 "msg"="collecting deployments" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:34:47.018090" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-116" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:34:47.540716" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-5dwqb4" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-03-09 10:34:48.067906" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-116" "deployment"="msi-deployment-116" "ts"="2026-03-09 10:34:48.560596" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-5dwqb4" "deployment"="customer-infra-basic-hcp-cluster-zsbbgv" "ts"="2026-03-09 10:34:49.634787" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:34:49.635001" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:34:49.635157" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:43:34.015898" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:43:34.015954" "level"=0 "msg"="deleting resource group" "resourceGroup"="basic-cluster-5dwqb4" "ts"="2026-03-09 10:45:03.662128" "level"=0 "msg"="finished deleting created resources" • [2055.843 seconds] ------------------------------ Ran 1 of 1 Specs in 2055.844 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:37 STEP: creating a resource group @ 03/09/26 10:10:47.922 "ts"="2026-03-09 10:10:47.922440" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-brcc4f" STEP: creating cluster parameters @ 03/09/26 10:10:51.961 STEP: creating customer resources @ 03/09/26 10:10:51.961 STEP: creating the HCP cluster @ 03/09/26 10:11:50.899 "ts"="2026-03-09 10:11:50.899550" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-brcc4f" STEP: creating a nodepool @ 03/09/26 10:20:06.162 STEP: attempting to update immutable platform profile fields @ 03/09/26 10:26:49.991 "ts"="2026-03-09 10:26:50.346326" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:50.346455" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-bkkmlm" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:26:51.662901" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-bkkmlm" "resourceGroup"="" "ts"="2026-03-09 10:26:52.574574" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-03-09 10:26:52.574691" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-03-09 10:26:52.574771" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:52.575907" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:26:52.575889" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-231" "ts"="2026-03-09 10:26:53.192223" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-brcc4f" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-03-09 10:26:53.724707" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-231" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:54.167175" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-brcc4f" "deployment"="customer-infra-negative-tests-cluster-bkkmlm" "ts"="2026-03-09 10:26:54.735792" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-231" "deployment"="msi-deployment-231" "ts"="2026-03-09 10:26:55.688377" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:55.688572" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:55.688746" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:38:00.981659" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:38:00.981723" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:39:26.956949" "level"=0 "msg"="finished deleting created resources" • [1719.057 seconds] ------------------------------ Ran 1 of 1 Specs in 1719.072 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:37 STEP: creating a resource group @ 03/09/26 10:10:47.922 "ts"="2026-03-09 10:10:47.922440" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-brcc4f" STEP: creating cluster parameters @ 03/09/26 10:10:51.961 STEP: creating customer resources @ 03/09/26 10:10:51.961 STEP: creating the HCP cluster @ 03/09/26 10:11:50.899 "ts"="2026-03-09 10:11:50.899550" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-brcc4f" STEP: creating a nodepool @ 03/09/26 10:20:06.162 STEP: attempting to update immutable platform profile fields @ 03/09/26 10:26:49.991 "ts"="2026-03-09 10:26:50.346326" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:50.346455" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-bkkmlm" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:26:51.662901" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-bkkmlm" "resourceGroup"="" "ts"="2026-03-09 10:26:52.574574" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-03-09 10:26:52.574691" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-03-09 10:26:52.574771" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:52.575907" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:26:52.575889" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-231" "ts"="2026-03-09 10:26:53.192223" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-brcc4f" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-03-09 10:26:53.724707" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-231" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:54.167175" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-brcc4f" "deployment"="customer-infra-negative-tests-cluster-bkkmlm" "ts"="2026-03-09 10:26:54.735792" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-231" "deployment"="msi-deployment-231" "ts"="2026-03-09 10:26:55.688377" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:55.688572" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:55.688746" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:38:00.981659" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:38:00.981723" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-negative-tests-brcc4f" "ts"="2026-03-09 10:39:26.956949" "level"=0 "msg"="finished deleting created resources" • [1719.057 seconds] ------------------------------ Ran 1 of 1 Specs in 1719.072 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create a cluster with an external auth config and get the external auth config [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:47 STEP: creating a resource group @ 03/09/26 10:10:47.865 "ts"="2026-03-09 10:10:47.866051" "level"=0 "msg"="creating resource group" "resourceGroup"="external-auth-cluster-4sjmt9" STEP: creating customer resources (infrastructure and managed identities) for cluster @ 03/09/26 10:10:52.16 STEP: creating the HCP cluster @ 03/09/26 10:11:50.705 "ts"="2026-03-09 10:11:50.705587" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ea-cluster" "resourceGroup"="external-auth-cluster-4sjmt9" STEP: getting credentials @ 03/09/26 10:19:35.761 STEP: ensuring the cluster is viable @ 03/09/26 10:20:07.832 STEP: creating the node pool @ 03/09/26 10:20:08.187 STEP: creating an app registration with a client secret @ 03/09/26 10:29:11.946 "ts"="2026-03-09 10:29:11.946635" "level"=0 "msg"="creating app registration" "appName"="aro-hcp-e2e-8562199231633121535" STEP: creating an external auth config with a prefix @ 03/09/26 10:29:16.158 STEP: verifying ExternalAuth is in a Succeeded state @ 03/09/26 10:29:26.78 STEP: creating a cluster role binding for the entra application @ 03/09/26 10:29:26.868 STEP: creating a rest config using OIDC authentication @ 03/09/26 10:29:27.121 STEP: confirming we can list namespaces using entra OIDC token @ 03/09/26 10:29:27.388 STEP: creating the console OAuth client secret for external auth @ 03/09/26 10:31:51.388 STEP: verifying cluster operators are available after external auth config creation @ 03/09/26 10:31:51.659 "ts"="2026-03-09 10:31:51.825588" "level"=0 "msg"="ClusterOperators not yet ready." "unavailable"=["console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown)"] "ts"="2026-03-09 10:32:11.826136" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:32:11.826323" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ea-cluster-25zhkh" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:32:12.817552" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ea-cluster-25zhkh" "resourceGroup"="" "ts"="2026-03-09 10:32:13.592596" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-5cf63771ae62e0f954d80324422e91fa0bb45dd988e9775341d966a9f013200c.yaml.gz" "ts"="2026-03-09 10:32:13.592727" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-5cf63771ae62e0f954d80324422e91fa0bb45dd988e9775341d966a9f013200c.yaml.gz" "ts"="2026-03-09 10:32:13.592807" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:32:13.594091" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-248" "ts"="2026-03-09 10:32:13.594154" "level"=0 "msg"="collecting deployments" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:32:14.215434" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-248" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:32:14.664526" "level"=0 "msg"="collecting operations" "resourceGroup"="external-auth-cluster-4sjmt9" "deployment"="pooledNonMsiScopedAssignments-ltdphft7fgrwg" "ts"="2026-03-09 10:32:15.151040" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-248" "deployment"="msi-deployment-248" "ts"="2026-03-09 10:32:15.575662" "level"=0 "msg"="collecting operations" "resourceGroup"="external-auth-cluster-4sjmt9" "deployment"="customer-infra-ea-cluster-25zhkh" "ts"="2026-03-09 10:32:16.542374" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:32:16.542602" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:32:16.542822" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:42:41.717272" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:42:41.717338" "level"=0 "msg"="deleting resource group" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:44:08.504987" "level"=0 "msg"="finished deleting created resources" • [2000.673 seconds] ------------------------------ Ran 1 of 1 Specs in 2000.673 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create a cluster with an external auth config and get the external auth config [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:47 STEP: creating a resource group @ 03/09/26 10:10:47.865 "ts"="2026-03-09 10:10:47.866051" "level"=0 "msg"="creating resource group" "resourceGroup"="external-auth-cluster-4sjmt9" STEP: creating customer resources (infrastructure and managed identities) for cluster @ 03/09/26 10:10:52.16 STEP: creating the HCP cluster @ 03/09/26 10:11:50.705 "ts"="2026-03-09 10:11:50.705587" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ea-cluster" "resourceGroup"="external-auth-cluster-4sjmt9" STEP: getting credentials @ 03/09/26 10:19:35.761 STEP: ensuring the cluster is viable @ 03/09/26 10:20:07.832 STEP: creating the node pool @ 03/09/26 10:20:08.187 STEP: creating an app registration with a client secret @ 03/09/26 10:29:11.946 "ts"="2026-03-09 10:29:11.946635" "level"=0 "msg"="creating app registration" "appName"="aro-hcp-e2e-8562199231633121535" STEP: creating an external auth config with a prefix @ 03/09/26 10:29:16.158 STEP: verifying ExternalAuth is in a Succeeded state @ 03/09/26 10:29:26.78 STEP: creating a cluster role binding for the entra application @ 03/09/26 10:29:26.868 STEP: creating a rest config using OIDC authentication @ 03/09/26 10:29:27.121 STEP: confirming we can list namespaces using entra OIDC token @ 03/09/26 10:29:27.388 STEP: creating the console OAuth client secret for external auth @ 03/09/26 10:31:51.388 STEP: verifying cluster operators are available after external auth config creation @ 03/09/26 10:31:51.659 "ts"="2026-03-09 10:31:51.825588" "level"=0 "msg"="ClusterOperators not yet ready." "unavailable"=["console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown)"] "ts"="2026-03-09 10:32:11.826136" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:32:11.826323" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ea-cluster-25zhkh" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:32:12.817552" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ea-cluster-25zhkh" "resourceGroup"="" "ts"="2026-03-09 10:32:13.592596" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-5cf63771ae62e0f954d80324422e91fa0bb45dd988e9775341d966a9f013200c.yaml.gz" "ts"="2026-03-09 10:32:13.592727" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-5cf63771ae62e0f954d80324422e91fa0bb45dd988e9775341d966a9f013200c.yaml.gz" "ts"="2026-03-09 10:32:13.592807" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:32:13.594091" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-248" "ts"="2026-03-09 10:32:13.594154" "level"=0 "msg"="collecting deployments" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:32:14.215434" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-248" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:32:14.664526" "level"=0 "msg"="collecting operations" "resourceGroup"="external-auth-cluster-4sjmt9" "deployment"="pooledNonMsiScopedAssignments-ltdphft7fgrwg" "ts"="2026-03-09 10:32:15.151040" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-248" "deployment"="msi-deployment-248" "ts"="2026-03-09 10:32:15.575662" "level"=0 "msg"="collecting operations" "resourceGroup"="external-auth-cluster-4sjmt9" "deployment"="customer-infra-ea-cluster-25zhkh" "ts"="2026-03-09 10:32:16.542374" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:32:16.542602" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:32:16.542822" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:42:41.717272" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:42:41.717338" "level"=0 "msg"="deleting resource group" "resourceGroup"="external-auth-cluster-4sjmt9" "ts"="2026-03-09 10:44:08.504987" "level"=0 "msg"="finished deleting created resources" • [2000.673 seconds] ------------------------------ Ran 1 of 1 Specs in 2000.673 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure ed resource group still exists @ 03/09/26 10:20:11.592 "ts"="2026-03-09 10:20:12.407489" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:20:12.407602" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-swqjnv" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:13.020583" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-swqjnv" "resourceGroup"="" "ts"="2026-03-09 10:20:13.581140" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-4zdhql" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:13.718642" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-4zdhql" "resourceGroup"="" "ts"="2026-03-09 10:20:15.009391" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster3-bjgscr" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:15.116323" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster3-bjgscr" "resourceGroup"="" "ts"="2026-03-09 10:20:15.881871" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-09 10:20:15.881997" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-09 10:20:15.882104" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:20:15.883018" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-137" "ts"="2026-03-09 10:20:15.883087" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "ts"="2026-03-09 10:20:15.883235" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-101" "ts"="2026-03-09 10:20:15.883234" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:16.578570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-137" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:16.989420" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:17.533363" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-101" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:18.050729" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="pooledNonMsiScopedAssignments-igpwm4vimfr3y" "ts"="2026-03-09 10:20:18.461745" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-137" "deployment"="msi-deployment-137" "ts"="2026-03-09 10:20:18.943848" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="msi-deployment-1" "ts"="2026-03-09 10:20:19.413908" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-101" "deployment"="msi-deployment-101" "ts"="2026-03-09 10:20:19.998242" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="customer-infra-basic-hcp-cluster3-bjgscr" "ts"="2026-03-09 10:20:23.020639" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-03-09 10:20:24.079779" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="customer-infra-basic-hcp-cluster2-4zdhql" "ts"="2026-03-09 10:20:25.089378" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-03-09 10:20:26.112562" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="customer-infra-basic-hcp-cluster-swqjnv" "ts"="2026-03-09 10:20:27.142024" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:20:27.142351" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:20:27.142586" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:29:02.517551" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:29:02.517609" "level"=0 "msg"="deleting resource group" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:30:58.455642" "level"=0 "msg"="finished deleting created resources" • [1210.618 seconds] ------------------------------ Ran 1 of 1 Specs in 1210.619 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') ed resource group still exists @ 03/09/26 10:20:11.592 "ts"="2026-03-09 10:20:12.407489" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:20:12.407602" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-swqjnv" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:13.020583" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-swqjnv" "resourceGroup"="" "ts"="2026-03-09 10:20:13.581140" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-4zdhql" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:13.718642" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-4zdhql" "resourceGroup"="" "ts"="2026-03-09 10:20:15.009391" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster3-bjgscr" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:15.116323" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster3-bjgscr" "resourceGroup"="" "ts"="2026-03-09 10:20:15.881871" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-09 10:20:15.881997" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-09 10:20:15.882104" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:20:15.883018" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-137" "ts"="2026-03-09 10:20:15.883087" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "ts"="2026-03-09 10:20:15.883235" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-101" "ts"="2026-03-09 10:20:15.883234" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:20:16.578570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-137" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:16.989420" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:17.533363" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-101" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:18.050729" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="pooledNonMsiScopedAssignments-igpwm4vimfr3y" "ts"="2026-03-09 10:20:18.461745" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-137" "deployment"="msi-deployment-137" "ts"="2026-03-09 10:20:18.943848" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="msi-deployment-1" "ts"="2026-03-09 10:20:19.413908" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-101" "deployment"="msi-deployment-101" "ts"="2026-03-09 10:20:19.998242" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="customer-infra-basic-hcp-cluster3-bjgscr" "ts"="2026-03-09 10:20:23.020639" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-03-09 10:20:24.079779" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="customer-infra-basic-hcp-cluster2-4zdhql" "ts"="2026-03-09 10:20:25.089378" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-03-09 10:20:26.112562" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-4lz8v2" "deployment"="customer-infra-basic-hcp-cluster-swqjnv" "ts"="2026-03-09 10:20:27.142024" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:20:27.142351" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:20:27.142586" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:29:02.517551" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:29:02.517609" "level"=0 "msg"="deleting resource group" "resourceGroup"="customer-rg-4lz8v2" "ts"="2026-03-09 10:30:58.455642" "level"=0 "msg"="finished deleting created resources" • [1210.618 seconds] ------------------------------ Ran 1 of 1 Specs in 1210.619 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36 STEP: creating a resource group @ 03/09/26 10:10:47.93 "ts"="2026-03-09 10:10:47.930589" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-bhcdxt" STEP: creating cluster parameters @ 03/09/26 10:10:51.915 STEP: creating customer resources @ 03/09/26 10:10:51.915 STEP: creating the HCP cluster @ 03/09/26 10:11:49.704 "ts"="2026-03-09 10:11:49.705076" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-bhcdxt" STEP: creating nodepool parameters with autoscaling enabled @ 03/09/26 10:19:23.725 STEP: creating the autoscaling nodepool @ 03/09/26 10:19:23.725 STEP: verifying the cluster has default autoscaling parameters @ 03/09/26 10:26:16.76 "ts"="2026-03-09 10:26:16.864222" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:16.864347" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-mlfc64" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:26:17.947876" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-mlfc64" "resourceGroup"="" "ts"="2026-03-09 10:26:18.850957" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz" "ts"="2026-03-09 10:26:18.851078" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz" "ts"="2026-03-09 10:26:18.851173" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:18.852156" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-289" "ts"="2026-03-09 10:26:18.852285" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:26:19.507463" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-289" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:19.954645" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-bhcdxt" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs" "ts"="2026-03-09 10:26:20.458134" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-289" "deployment"="msi-deployment-289" "ts"="2026-03-09 10:26:20.965146" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-bhcdxt" "deployment"="customer-infra-np-autoscale-cluster-mlfc64" "ts"="2026-03-09 10:26:21.938401" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:21.938568" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:21.938712" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:36:16.972876" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:36:16.972932" "level"=0 "msg"="deleting resource group" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:38:02.868147" "level"=0 "msg"="finished deleting created resources" • [1634.994 seconds] ------------------------------ Ran 1 of 1 Specs in 1634.994 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36 STEP: creating a resource group @ 03/09/26 10:10:47.93 "ts"="2026-03-09 10:10:47.930589" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-bhcdxt" STEP: creating cluster parameters @ 03/09/26 10:10:51.915 STEP: creating customer resources @ 03/09/26 10:10:51.915 STEP: creating the HCP cluster @ 03/09/26 10:11:49.704 "ts"="2026-03-09 10:11:49.705076" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-bhcdxt" STEP: creating nodepool parameters with autoscaling enabled @ 03/09/26 10:19:23.725 STEP: creating the autoscaling nodepool @ 03/09/26 10:19:23.725 STEP: verifying the cluster has default autoscaling parameters @ 03/09/26 10:26:16.76 "ts"="2026-03-09 10:26:16.864222" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:16.864347" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-mlfc64" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:26:17.947876" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-mlfc64" "resourceGroup"="" "ts"="2026-03-09 10:26:18.850957" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz" "ts"="2026-03-09 10:26:18.851078" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz" "ts"="2026-03-09 10:26:18.851173" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:18.852156" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-289" "ts"="2026-03-09 10:26:18.852285" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:26:19.507463" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-289" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:19.954645" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-bhcdxt" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs" "ts"="2026-03-09 10:26:20.458134" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-289" "deployment"="msi-deployment-289" "ts"="2026-03-09 10:26:20.965146" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-bhcdxt" "deployment"="customer-infra-np-autoscale-cluster-mlfc64" "ts"="2026-03-09 10:26:21.938401" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:21.938568" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:21.938712" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:36:16.972876" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:36:16.972932" "level"=0 "msg"="deleting resource group" "resourceGroup"="np-autoscaling-bhcdxt" "ts"="2026-03-09 10:38:02.868147" "level"=0 "msg"="finished deleting created resources" • [1634.994 seconds] ------------------------------ Ran 1 of 1 Specs in 1634.994 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure o/runtimeclasses; list admissionregistration.k8s.io/mutatingwebhookconfigurations; list admissionregistration.k8s.io/validatingwebhookconfigurations; list apiregistration.k8s.io/apiservices; list config.openshift.io/clusterversions; list config.openshift.io/clusteroperators; list config.openshift.io/infrastructures; list monitoring.coreos.com/prometheusrules; list monitoring.coreos.com/servicemonitors; list machine.openshift.io/machines; list machine.openshift.io/machinesets; list operator.openshift.io/ingresscontrollers; list security.openshift.io/securitycontextconstraints; list route.openshift.io/routes; list image.openshift.io/imagestreams; create authentication.k8s.io/tokenreviews; create authorization.k8s.io/selfsubjectaccessreviews; create authorization.k8s.io/subjectaccessreviews) @ 03/09/26 10:20:30.497 STEP: verifying VerifyListNamespaced(namespace=kube-system, resources=secrets) @ 03/09/26 10:20:38.951 STEP: verifying VerifyRBACAllowed(get /secrets; create /pods; delete /pods; create /namespaces; create apps/deployments; delete apps/deployments; update apps/deployments) @ 03/09/26 10:20:39.013 STEP: waiting for the session to expire @ 03/09/26 10:20:39.384 STEP: verifying the session is expired @ 03/09/26 10:22:24.005 STEP: trying to access a breakglass session of another user @ 03/09/26 10:22:24.23 STEP: reaching out to the admin API to create a breakglass session for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-hzp78t/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster with aro-sre-pso permissions: https://admin.j2548992.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-hzp78t/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster/breakglass @ 03/09/26 10:22:24.23 STEP: waiting for SRE breakglass session to be ready at /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-hzp78t/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-7jk85/kubeconfig @ 03/09/26 10:22:24.568 STEP: and expecting cluster access to be denied @ 03/09/26 10:22:29.624 "ts"="2026-03-09 10:22:30.055993" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:22:30.056109" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-sre-hcp-cluster-c7h97c" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:22:31.233312" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-sre-hcp-cluster-c7h97c" "resourceGroup"="" "ts"="2026-03-09 10:22:32.033391" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-c550bd9856d8576e40dce24068d9220b0bd5f3dcb700639debaf3003763a0444.yaml.gz" "ts"="2026-03-09 10:22:32.033526" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-c550bd9856d8576e40dce24068d9220b0bd5f3dcb700639debaf3003763a0444.yaml.gz" "ts"="2026-03-09 10:22:32.033623" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:22:32.034571" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-45" "ts"="2026-03-09 10:22:32.034614" "level"=0 "msg"="collecting deployments" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:22:32.629929" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-45" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:22:33.198337" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-api-breakglass-hzp78t" "deployment"="pooledNonMsiScopedAssignments-456d7mdgworfy" "ts"="2026-03-09 10:22:33.667913" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-45" "deployment"="msi-deployment-45" "ts"="2026-03-09 10:22:34.360873" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-api-breakglass-hzp78t" "deployment"="customer-infra-sre-hcp-cluster-c7h97c" "ts"="2026-03-09 10:22:35.319011" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:22:35.319243" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:22:35.319464" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:30:59.869409" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:30:59.869473" "level"=0 "msg"="deleting resource group" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:32:28.072323" "level"=0 "msg"="finished deleting created resources" • [1300.279 seconds] ------------------------------ Ran 1 of 1 Specs in 1300.279 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') o/runtimeclasses; list admissionregistration.k8s.io/mutatingwebhookconfigurations; list admissionregistration.k8s.io/validatingwebhookconfigurations; list apiregistration.k8s.io/apiservices; list config.openshift.io/clusterversions; list config.openshift.io/clusteroperators; list config.openshift.io/infrastructures; list monitoring.coreos.com/prometheusrules; list monitoring.coreos.com/servicemonitors; list machine.openshift.io/machines; list machine.openshift.io/machinesets; list operator.openshift.io/ingresscontrollers; list security.openshift.io/securitycontextconstraints; list route.openshift.io/routes; list image.openshift.io/imagestreams; create authentication.k8s.io/tokenreviews; create authorization.k8s.io/selfsubjectaccessreviews; create authorization.k8s.io/subjectaccessreviews) @ 03/09/26 10:20:30.497 STEP: verifying VerifyListNamespaced(namespace=kube-system, resources=secrets) @ 03/09/26 10:20:38.951 STEP: verifying VerifyRBACAllowed(get /secrets; create /pods; delete /pods; create /namespaces; create apps/deployments; delete apps/deployments; update apps/deployments) @ 03/09/26 10:20:39.013 STEP: waiting for the session to expire @ 03/09/26 10:20:39.384 STEP: verifying the session is expired @ 03/09/26 10:22:24.005 STEP: trying to access a breakglass session of another user @ 03/09/26 10:22:24.23 STEP: reaching out to the admin API to create a breakglass session for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-hzp78t/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster with aro-sre-pso permissions: https://admin.j2548992.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-hzp78t/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster/breakglass @ 03/09/26 10:22:24.23 STEP: waiting for SRE breakglass session to be ready at /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-hzp78t/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-7jk85/kubeconfig @ 03/09/26 10:22:24.568 STEP: and expecting cluster access to be denied @ 03/09/26 10:22:29.624 "ts"="2026-03-09 10:22:30.055993" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:22:30.056109" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-sre-hcp-cluster-c7h97c" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:22:31.233312" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-sre-hcp-cluster-c7h97c" "resourceGroup"="" "ts"="2026-03-09 10:22:32.033391" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-c550bd9856d8576e40dce24068d9220b0bd5f3dcb700639debaf3003763a0444.yaml.gz" "ts"="2026-03-09 10:22:32.033526" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-c550bd9856d8576e40dce24068d9220b0bd5f3dcb700639debaf3003763a0444.yaml.gz" "ts"="2026-03-09 10:22:32.033623" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:22:32.034571" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-45" "ts"="2026-03-09 10:22:32.034614" "level"=0 "msg"="collecting deployments" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:22:32.629929" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-45" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:22:33.198337" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-api-breakglass-hzp78t" "deployment"="pooledNonMsiScopedAssignments-456d7mdgworfy" "ts"="2026-03-09 10:22:33.667913" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-45" "deployment"="msi-deployment-45" "ts"="2026-03-09 10:22:34.360873" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-api-breakglass-hzp78t" "deployment"="customer-infra-sre-hcp-cluster-c7h97c" "ts"="2026-03-09 10:22:35.319011" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:22:35.319243" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:22:35.319464" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:30:59.869409" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:30:59.869473" "level"=0 "msg"="deleting resource group" "resourceGroup"="admin-api-breakglass-hzp78t" "ts"="2026-03-09 10:32:28.072323" "level"=0 "msg"="finished deleting created resources" • [1300.279 seconds] ------------------------------ Ran 1 of 1 Specs in 1300.279 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:120 STEP: creating a resource group @ 03/09/26 10:30:58.86 "ts"="2026-03-09 10:30:58.860624" "level"=0 "msg"="creating resource group" "resourceGroup"="patch-tags-pxqr4n" STEP: creating cluster parameters @ 03/09/26 10:31:01.856 STEP: creating customer resources @ 03/09/26 10:31:01.856 STEP: creating the HCP cluster @ 03/09/26 10:32:00.091 "ts"="2026-03-09 10:32:00.092051" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="patch-tags-cluster" "resourceGroup"="patch-tags-pxqr4n" STEP: getting credentials @ 03/09/26 10:38:03.464 STEP: ensuring the cluster is viable @ 03/09/26 10:38:54.6 STEP: sending a PATCH request to set a tag @ 03/09/26 10:38:55.028 STEP: verifying the tag is present in the update response body @ 03/09/26 10:39:35.857 STEP: verifying the tag is present on the cluster @ 03/09/26 10:39:35.857 "ts"="2026-03-09 10:39:35.967590" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:39:35.967743" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-tags-cluster-ddgqtt" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:39:36.966000" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-tags-cluster-ddgqtt" "resourceGroup"="" "ts"="2026-03-09 10:39:38.037822" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-03-09 10:39:38.037962" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-03-09 10:39:38.038046" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:39:38.039324" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "ts"="2026-03-09 10:39:38.039357" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:39:38.663283" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:39:39.092513" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-pxqr4n" "deployment"="pooledNonMsiScopedAssignments-fklo75htpxzxc" "ts"="2026-03-09 10:39:39.532248" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="msi-deployment-1" "ts"="2026-03-09 10:39:39.990307" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-pxqr4n" "deployment"="customer-infra-patch-tags-cluster-ddgqtt" "ts"="2026-03-09 10:39:40.959434" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:39:40.960089" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:39:40.960324" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:48:35.606888" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:48:35.606954" "level"=0 "msg"="deleting resource group" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:50:01.392331" "level"=0 "msg"="finished deleting created resources" • [2353.450 seconds] ------------------------------ Ran 1 of 1 Specs in 2353.451 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:120 STEP: creating a resource group @ 03/09/26 10:30:58.86 "ts"="2026-03-09 10:30:58.860624" "level"=0 "msg"="creating resource group" "resourceGroup"="patch-tags-pxqr4n" STEP: creating cluster parameters @ 03/09/26 10:31:01.856 STEP: creating customer resources @ 03/09/26 10:31:01.856 STEP: creating the HCP cluster @ 03/09/26 10:32:00.091 "ts"="2026-03-09 10:32:00.092051" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="patch-tags-cluster" "resourceGroup"="patch-tags-pxqr4n" STEP: getting credentials @ 03/09/26 10:38:03.464 STEP: ensuring the cluster is viable @ 03/09/26 10:38:54.6 STEP: sending a PATCH request to set a tag @ 03/09/26 10:38:55.028 STEP: verifying the tag is present in the update response body @ 03/09/26 10:39:35.857 STEP: verifying the tag is present on the cluster @ 03/09/26 10:39:35.857 "ts"="2026-03-09 10:39:35.967590" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:39:35.967743" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-tags-cluster-ddgqtt" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:39:36.966000" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-tags-cluster-ddgqtt" "resourceGroup"="" "ts"="2026-03-09 10:39:38.037822" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-03-09 10:39:38.037962" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-03-09 10:39:38.038046" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:39:38.039324" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "ts"="2026-03-09 10:39:38.039357" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:39:38.663283" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:39:39.092513" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-pxqr4n" "deployment"="pooledNonMsiScopedAssignments-fklo75htpxzxc" "ts"="2026-03-09 10:39:39.532248" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-1" "deployment"="msi-deployment-1" "ts"="2026-03-09 10:39:39.990307" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-pxqr4n" "deployment"="customer-infra-patch-tags-cluster-ddgqtt" "ts"="2026-03-09 10:39:40.959434" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:39:40.960089" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:39:40.960324" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:48:35.606888" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:48:35.606954" "level"=0 "msg"="deleting resource group" "resourceGroup"="patch-tags-pxqr4n" "ts"="2026-03-09 10:50:01.392331" "level"=0 "msg"="finished deleting created resources" • [2353.450 seconds] ------------------------------ Ran 1 of 1 Specs in 2353.451 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create node pools with ARM64-based VMs [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:35 "ts"="2026-03-09 10:10:47.801070" "level"=0 "msg"="initialized managed identities pool state" "entries"=20 STEP: creating a resource group @ 03/09/26 10:10:47.804 "ts"="2026-03-09 10:10:47.804626" "level"=0 "msg"="creating resource group" "resourceGroup"="arm64-vm-cluster-tg2jjn" STEP: creating cluster parameters @ 03/09/26 10:10:51.5 STEP: creating customer resources @ 03/09/26 10:10:51.501 STEP: creating the HCP cluster @ 03/09/26 10:11:49.665 "ts"="2026-03-09 10:11:49.665660" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="arm64-vm-hcp-cluster" "resourceGroup"="arm64-vm-cluster-tg2jjn" STEP: discovering an ARM64-capable VM size @ 03/09/26 10:18:24.169 STEP: creating the ARM64-based node pool "arm64-vm-np-1" with VM size "Standard_D2pls_v6" @ 03/09/26 10:18:25.11 "ts"="2026-03-09 10:25:19.915031" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:25:19.915176" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-arm64-vm-hcp-cluster-h89ts8" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:25:20.978917" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-arm64-vm-hcp-cluster-h89ts8" "resourceGroup"="" "ts"="2026-03-09 10:25:21.645813" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz" "ts"="2026-03-09 10:25:21.645947" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz" "ts"="2026-03-09 10:25:21.646032" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:25:21.647012" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-96" "ts"="2026-03-09 10:25:21.647064" "level"=0 "msg"="collecting deployments" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:25:22.263251" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-96" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:25:22.725482" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-tg2jjn" "deployment"="pooledNonMsiScopedAssignments-6htyoibevj7ye" "ts"="2026-03-09 10:25:23.180258" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-96" "deployment"="msi-deployment-96" "ts"="2026-03-09 10:25:23.676997" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-tg2jjn" "deployment"="customer-infra-arm64-vm-hcp-cluster-h89ts8" "ts"="2026-03-09 10:25:24.719240" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:25:24.719426" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:25:24.719569" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:34:08.979543" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:34:08.979599" "level"=0 "msg"="deleting resource group" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:35:39.294939" "level"=0 "msg"="finished deleting created resources" • [1491.509 seconds] ------------------------------ Ran 1 of 1 Specs in 1491.509 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create node pools with ARM64-based VMs [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:35 "ts"="2026-03-09 10:10:47.801070" "level"=0 "msg"="initialized managed identities pool state" "entries"=20 STEP: creating a resource group @ 03/09/26 10:10:47.804 "ts"="2026-03-09 10:10:47.804626" "level"=0 "msg"="creating resource group" "resourceGroup"="arm64-vm-cluster-tg2jjn" STEP: creating cluster parameters @ 03/09/26 10:10:51.5 STEP: creating customer resources @ 03/09/26 10:10:51.501 STEP: creating the HCP cluster @ 03/09/26 10:11:49.665 "ts"="2026-03-09 10:11:49.665660" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="arm64-vm-hcp-cluster" "resourceGroup"="arm64-vm-cluster-tg2jjn" STEP: discovering an ARM64-capable VM size @ 03/09/26 10:18:24.169 STEP: creating the ARM64-based node pool "arm64-vm-np-1" with VM size "Standard_D2pls_v6" @ 03/09/26 10:18:25.11 "ts"="2026-03-09 10:25:19.915031" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:25:19.915176" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-arm64-vm-hcp-cluster-h89ts8" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:25:20.978917" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-arm64-vm-hcp-cluster-h89ts8" "resourceGroup"="" "ts"="2026-03-09 10:25:21.645813" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz" "ts"="2026-03-09 10:25:21.645947" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz" "ts"="2026-03-09 10:25:21.646032" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:25:21.647012" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-96" "ts"="2026-03-09 10:25:21.647064" "level"=0 "msg"="collecting deployments" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:25:22.263251" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-96" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:25:22.725482" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-tg2jjn" "deployment"="pooledNonMsiScopedAssignments-6htyoibevj7ye" "ts"="2026-03-09 10:25:23.180258" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-96" "deployment"="msi-deployment-96" "ts"="2026-03-09 10:25:23.676997" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-tg2jjn" "deployment"="customer-infra-arm64-vm-hcp-cluster-h89ts8" "ts"="2026-03-09 10:25:24.719240" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:25:24.719426" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:25:24.719569" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:34:08.979543" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:34:08.979599" "level"=0 "msg"="deleting resource group" "resourceGroup"="arm64-vm-cluster-tg2jjn" "ts"="2026-03-09 10:35:39.294939" "level"=0 "msg"="finished deleting created resources" • [1491.509 seconds] ------------------------------ Ran 1 of 1 Specs in 1491.509 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster and custom node pool osDisk size [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:35 STEP: creating a resource group @ 03/09/26 10:30:58.75 "ts"="2026-03-09 10:30:58.750300" "level"=0 "msg"="creating resource group" "resourceGroup"="clusternp128-2fbz5q" STEP: creating customer resources (infrastructure and managed identities) for cluster @ 03/09/26 10:31:01.348 STEP: creating the HCP cluster @ 03/09/26 10:31:49.961 "ts"="2026-03-09 10:31:49.961726" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="hcp-cluster-np-128" "resourceGroup"="clusternp128-2fbz5q" STEP: creating the node pool with custom osDisk size @ 03/09/26 10:37:02.971 STEP: getting credentials @ 03/09/26 10:43:56.015 STEP: ensuring the cluster is viable @ 03/09/26 10:44:07.03 STEP: verifying the node pool is created and has the correct osDisk size @ 03/09/26 10:44:07.42 "ts"="2026-03-09 10:44:07.509589" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:44:07.509743" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-hcp-cluster-np-128-hhpt88" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:44:08.781608" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-hcp-cluster-np-128-hhpt88" "resourceGroup"="" "ts"="2026-03-09 10:44:09.576694" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz" "ts"="2026-03-09 10:44:09.577016" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz" "ts"="2026-03-09 10:44:09.577253" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:44:09.579335" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "ts"="2026-03-09 10:44:09.579396" "level"=0 "msg"="collecting deployments" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:44:10.410719" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:44:11.073515" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-2fbz5q" "deployment"="pooledNonMsiScopedAssignments-3ljyjjx2n7gxo" "ts"="2026-03-09 10:44:11.848324" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="msi-deployment-88" "ts"="2026-03-09 10:44:12.551465" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-2fbz5q" "deployment"="customer-infra-hcp-cluster-np-128-hhpt88" "ts"="2026-03-09 10:44:14.032306" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:44:14.032539" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:44:14.032833" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:54:29.109989" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:54:29.110058" "level"=0 "msg"="deleting resource group" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:55:55.441080" "level"=0 "msg"="finished deleting created resources" • [2707.511 seconds] ------------------------------ Ran 1 of 1 Specs in 2707.512 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster and custom node pool osDisk size [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:35 STEP: creating a resource group @ 03/09/26 10:30:58.75 "ts"="2026-03-09 10:30:58.750300" "level"=0 "msg"="creating resource group" "resourceGroup"="clusternp128-2fbz5q" STEP: creating customer resources (infrastructure and managed identities) for cluster @ 03/09/26 10:31:01.348 STEP: creating the HCP cluster @ 03/09/26 10:31:49.961 "ts"="2026-03-09 10:31:49.961726" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="hcp-cluster-np-128" "resourceGroup"="clusternp128-2fbz5q" STEP: creating the node pool with custom osDisk size @ 03/09/26 10:37:02.971 STEP: getting credentials @ 03/09/26 10:43:56.015 STEP: ensuring the cluster is viable @ 03/09/26 10:44:07.03 STEP: verifying the node pool is created and has the correct osDisk size @ 03/09/26 10:44:07.42 "ts"="2026-03-09 10:44:07.509589" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:44:07.509743" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-hcp-cluster-np-128-hhpt88" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:44:08.781608" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-hcp-cluster-np-128-hhpt88" "resourceGroup"="" "ts"="2026-03-09 10:44:09.576694" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz" "ts"="2026-03-09 10:44:09.577016" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz" "ts"="2026-03-09 10:44:09.577253" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:44:09.579335" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "ts"="2026-03-09 10:44:09.579396" "level"=0 "msg"="collecting deployments" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:44:10.410719" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:44:11.073515" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-2fbz5q" "deployment"="pooledNonMsiScopedAssignments-3ljyjjx2n7gxo" "ts"="2026-03-09 10:44:11.848324" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="msi-deployment-88" "ts"="2026-03-09 10:44:12.551465" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-2fbz5q" "deployment"="customer-infra-hcp-cluster-np-128-hhpt88" "ts"="2026-03-09 10:44:14.032306" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:44:14.032539" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:44:14.032833" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:54:29.109989" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:54:29.110058" "level"=0 "msg"="deleting resource group" "resourceGroup"="clusternp128-2fbz5q" "ts"="2026-03-09 10:55:55.441080" "level"=0 "msg"="finished deleting created resources" • [2707.511 seconds] ------------------------------ Ran 1 of 1 Specs in 2707.512 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34 STEP: creating a resource group @ 03/09/26 10:10:47.852 "ts"="2026-03-09 10:10:47.853001" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-krpv6r" STEP: creating cluster parameters @ 03/09/26 10:10:51.766 STEP: creating customer resources @ 03/09/26 10:10:51.766 STEP: creating the HCP cluster @ 03/09/26 10:11:49.181 "ts"="2026-03-09 10:11:49.181764" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-krpv6r" STEP: verifying kusto logs are present @ 03/09/26 10:18:22.816 "ts"="2026-03-09 10:18:31.668019" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:18:31.668366" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-engineering-hcp-cluster-pdnjqw" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:18:32.992820" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-engineering-hcp-cluster-pdnjqw" "resourceGroup"="" "ts"="2026-03-09 10:18:33.760631" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz" "ts"="2026-03-09 10:18:33.760894" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz" "ts"="2026-03-09 10:18:33.761243" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:18:33.762830" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "ts"="2026-03-09 10:18:33.762893" "level"=0 "msg"="collecting deployments" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:18:34.562574" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-krpv6r" "deployment"="pooledNonMsiScopedAssignments-5borhjiq7mwd4" "ts"="2026-03-09 10:18:35.205550" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:35.863235" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-krpv6r" "deployment"="customer-infra-engineering-hcp-cluster-pdnjqw" "ts"="2026-03-09 10:18:36.487283" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="msi-deployment-88" "ts"="2026-03-09 10:18:37.737071" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:18:37.737270" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:18:37.737436" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:29:02.926275" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:29:02.926358" "level"=0 "msg"="deleting resource group" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:30:40.515645" "level"=0 "msg"="finished deleting created resources" • [1192.687 seconds] ------------------------------ Ran 1 of 1 Specs in 1192.688 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34 STEP: creating a resource group @ 03/09/26 10:10:47.852 "ts"="2026-03-09 10:10:47.853001" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-krpv6r" STEP: creating cluster parameters @ 03/09/26 10:10:51.766 STEP: creating customer resources @ 03/09/26 10:10:51.766 STEP: creating the HCP cluster @ 03/09/26 10:11:49.181 "ts"="2026-03-09 10:11:49.181764" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-krpv6r" STEP: verifying kusto logs are present @ 03/09/26 10:18:22.816 "ts"="2026-03-09 10:18:31.668019" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:18:31.668366" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-engineering-hcp-cluster-pdnjqw" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:18:32.992820" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-engineering-hcp-cluster-pdnjqw" "resourceGroup"="" "ts"="2026-03-09 10:18:33.760631" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz" "ts"="2026-03-09 10:18:33.760894" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz" "ts"="2026-03-09 10:18:33.761243" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:18:33.762830" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "ts"="2026-03-09 10:18:33.762893" "level"=0 "msg"="collecting deployments" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:18:34.562574" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-krpv6r" "deployment"="pooledNonMsiScopedAssignments-5borhjiq7mwd4" "ts"="2026-03-09 10:18:35.205550" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:35.863235" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-krpv6r" "deployment"="customer-infra-engineering-hcp-cluster-pdnjqw" "ts"="2026-03-09 10:18:36.487283" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-88" "deployment"="msi-deployment-88" "ts"="2026-03-09 10:18:37.737071" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:18:37.737270" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:18:37.737436" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:29:02.926275" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:29:02.926358" "level"=0 "msg"="deleting resource group" "resourceGroup"="kusto-logs-krpv6r" "ts"="2026-03-09 10:30:40.515645" "level"=0 "msg"="finished deleting created resources" • [1192.687 seconds] ------------------------------ Ran 1 of 1 Specs in 1192.688 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel Show raw failure : creating cluster without node pool: list-test-cluster-x5g4wb @ 03/09/26 10:37:05.684 STEP: creating customer resources for cluster: list-test-cluster-x5g4wb @ 03/09/26 10:37:05.684 STEP: creating HCP cluster: list-test-cluster-x5g4wb @ 03/09/26 10:38:20.891 "ts"="2026-03-09 10:38:20.891083" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="list-test-cluster-x5g4wb" "resourceGroup"="cluster-listing-tj52tb" STEP: testing subscription-level cluster listing @ 03/09/26 10:44:23.565 STEP: testing resource group-level cluster listing @ 03/09/26 10:44:23.944 "ts"="2026-03-09 10:44:24.191537" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:44:24.191706" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-list-test-cluster-ptq6ns-sfxgf7" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:44:25.481099" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-list-test-cluster-ptq6ns-sfxgf7" "resourceGroup"="" "ts"="2026-03-09 10:44:26.684889" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-list-test-cluster-x5g4wb-lb2rp8" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:44:26.885748" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-list-test-cluster-x5g4wb-lb2rp8" "resourceGroup"="" "ts"="2026-03-09 10:44:27.498630" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b860a86958f57179c37d87e5f83be5384f83e5bf95c88d961f55ca4e4ca1bbbb.yaml.gz" "ts"="2026-03-09 10:44:27.498776" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b860a86958f57179c37d87e5f83be5384f83e5bf95c88d961f55ca4e4ca1bbbb.yaml.gz" "ts"="2026-03-09 10:44:27.498903" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:44:27.500338" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "ts"="2026-03-09 10:44:27.500424" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "ts"="2026-03-09 10:44:27.500408" "level"=0 "msg"="collecting deployments" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:44:27.500483" "level"=0 "msg"="collecting deployments" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:44:28.189501" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:44:28.717203" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:44:29.238430" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-tj52tb" "deployment"="pooledNonMsiScopedAssignments-jbjqdczfb6u6g" "ts"="2026-03-09 10:44:29.821004" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-sztzt2" "deployment"="pooledNonMsiScopedAssignments-u4p7fc4nt5fwe" "ts"="2026-03-09 10:44:30.412003" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="msi-deployment-149" "ts"="2026-03-09 10:44:31.005516" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="msi-deployment-98" "ts"="2026-03-09 10:44:31.556001" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-tj52tb" "deployment"="customer-infra-list-test-cluster-x5g4wb-lb2rp8" "ts"="2026-03-09 10:44:32.139686" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-sztzt2" "deployment"="customer-infra-list-test-cluster-ptq6ns-sfxgf7" "ts"="2026-03-09 10:44:34.620529" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:44:34.620761" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:44:34.621014" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:44:34.621129" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:53:20.240643" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:53:20.240703" "level"=0 "msg"="deleting resource group" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:54:29.517335" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:54:29.517383" "level"=0 "msg"="deleting resource group" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:56:06.363205" "level"=0 "msg"="finished deleting created resources" • [2718.487 seconds] ------------------------------ Ran 1 of 1 Specs in 2718.487 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') : creating cluster without node pool: list-test-cluster-x5g4wb @ 03/09/26 10:37:05.684 STEP: creating customer resources for cluster: list-test-cluster-x5g4wb @ 03/09/26 10:37:05.684 STEP: creating HCP cluster: list-test-cluster-x5g4wb @ 03/09/26 10:38:20.891 "ts"="2026-03-09 10:38:20.891083" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="list-test-cluster-x5g4wb" "resourceGroup"="cluster-listing-tj52tb" STEP: testing subscription-level cluster listing @ 03/09/26 10:44:23.565 STEP: testing resource group-level cluster listing @ 03/09/26 10:44:23.944 "ts"="2026-03-09 10:44:24.191537" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:44:24.191706" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-list-test-cluster-ptq6ns-sfxgf7" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:44:25.481099" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-list-test-cluster-ptq6ns-sfxgf7" "resourceGroup"="" "ts"="2026-03-09 10:44:26.684889" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-list-test-cluster-x5g4wb-lb2rp8" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:44:26.885748" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-list-test-cluster-x5g4wb-lb2rp8" "resourceGroup"="" "ts"="2026-03-09 10:44:27.498630" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b860a86958f57179c37d87e5f83be5384f83e5bf95c88d961f55ca4e4ca1bbbb.yaml.gz" "ts"="2026-03-09 10:44:27.498776" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b860a86958f57179c37d87e5f83be5384f83e5bf95c88d961f55ca4e4ca1bbbb.yaml.gz" "ts"="2026-03-09 10:44:27.498903" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:44:27.500338" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "ts"="2026-03-09 10:44:27.500424" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "ts"="2026-03-09 10:44:27.500408" "level"=0 "msg"="collecting deployments" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:44:27.500483" "level"=0 "msg"="collecting deployments" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:44:28.189501" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:44:28.717203" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:44:29.238430" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-tj52tb" "deployment"="pooledNonMsiScopedAssignments-jbjqdczfb6u6g" "ts"="2026-03-09 10:44:29.821004" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-sztzt2" "deployment"="pooledNonMsiScopedAssignments-u4p7fc4nt5fwe" "ts"="2026-03-09 10:44:30.412003" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="msi-deployment-149" "ts"="2026-03-09 10:44:31.005516" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="msi-deployment-98" "ts"="2026-03-09 10:44:31.556001" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-tj52tb" "deployment"="customer-infra-list-test-cluster-x5g4wb-lb2rp8" "ts"="2026-03-09 10:44:32.139686" "level"=0 "msg"="collecting operations" "resourceGroup"="cluster-listing-sztzt2" "deployment"="customer-infra-list-test-cluster-ptq6ns-sfxgf7" "ts"="2026-03-09 10:44:34.620529" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:44:34.620761" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:44:34.621014" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:44:34.621129" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:53:20.240643" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:53:20.240703" "level"=0 "msg"="deleting resource group" "resourceGroup"="cluster-listing-tj52tb" "ts"="2026-03-09 10:54:29.517335" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:54:29.517383" "level"=0 "msg"="deleting resource group" "resourceGroup"="cluster-listing-sztzt2" "ts"="2026-03-09 10:56:06.363205" "level"=0 "msg"="finished deleting created resources" • [2718.487 seconds] ------------------------------ Ran 1 of 1 Specs in 2718.487 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure plan phase is \"Installing\", expected Complete\nNo InstallPlan conditions available" STEP: creating NodeFeatureDiscovery CR to deploy NFD worker @ 03/09/26 10:31:05.782 STEP: waiting for NFD worker DaemonSet to be created @ 03/09/26 10:31:05.853 STEP: waiting for NFD worker pods to be created and verify images from registry.redhat.io can be pulled @ 03/09/26 10:31:20.961 "ts"="2026-03-09 10:31:20.961556" "level"=0 "msg"="Starting image pull verification" "namespace"="openshift-nfd" "imageRepository"="registry.redhat.io" "imageName"="ose-node-feature-discovery" "startTime"="2026-03-09T10:31:20Z" "ts"="2026-03-09 10:31:21.063281" "level"=0 "msg"="Container waiting" "pod"="nfd-gc-685d666965-pmgj7" "container"="nfd-gc" "reason"="CreateContainerError" "message"="container create failed: executable file `nfd-gc` not found in $PATH: No such file or directory\n" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "ts"="2026-03-09 10:31:21.063340" "level"=0 "msg"="Successfully pulled image" "pod"="nfd-master-59c77c96f9-mf5fm" "container"="nfd-master" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "imageID"="registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:57a132bdf888ea95345027876c952d09ad72f43a5b4488794bf4f6c45f8520da" "ts"="2026-03-09 10:31:21.063367" "level"=0 "msg"="Successfully pulled image" "pod"="nfd-worker-g5nvf" "container"="nfd-worker" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "imageID"="registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:57a132bdf888ea95345027876c952d09ad72f43a5b4488794bf4f6c45f8520da" "ts"="2026-03-09 10:31:21.063404" "level"=0 "msg"="Successfully pulled image" "pod"="nfd-worker-z4lcj" "container"="nfd-worker" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "imageID"="registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:57a132bdf888ea95345027876c952d09ad72f43a5b4488794bf4f6c45f8520da" "ts"="2026-03-09 10:31:21.063414" "level"=0 "msg"="Image pull verification completed successfully" "namespace"="openshift-nfd" "imageRepository"="registry.redhat.io" "imageName"="ose-node-feature-discovery" "duration"="101.862899ms" "endTime"="2026-03-09T10:31:21Z" "ts"="2026-03-09 10:31:21.063598" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:31:21.063711" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-pullsecret-hcp-cluster-pnm7dk" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:31:22.198453" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-pullsecret-hcp-cluster-pnm7dk" "resourceGroup"="" "ts"="2026-03-09 10:31:22.797001" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz" "ts"="2026-03-09 10:31:22.797202" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz" "ts"="2026-03-09 10:31:22.797291" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:31:22.798495" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-111" "ts"="2026-03-09 10:31:22.798537" "level"=0 "msg"="collecting deployments" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:31:23.544608" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-111" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:31:23.994258" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-4w8kdv" "deployment"="pooledNonMsiScopedAssignments-p2h32vyw2l2ma" "ts"="2026-03-09 10:31:24.616274" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-111" "deployment"="msi-deployment-111" "ts"="2026-03-09 10:31:25.197957" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-4w8kdv" "deployment"="customer-infra-pullsecret-hcp-cluster-pnm7dk" "ts"="2026-03-09 10:31:26.294381" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:31:26.294584" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:31:26.294775" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:40:41.310398" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:40:41.310472" "level"=0 "msg"="deleting resource group" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:41:53.715248" "level"=0 "msg"="finished deleting created resources" • [1865.902 seconds] ------------------------------ Ran 1 of 1 Specs in 1865.903 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') plan phase is \"Installing\", expected Complete\nNo InstallPlan conditions available" STEP: creating NodeFeatureDiscovery CR to deploy NFD worker @ 03/09/26 10:31:05.782 STEP: waiting for NFD worker DaemonSet to be created @ 03/09/26 10:31:05.853 STEP: waiting for NFD worker pods to be created and verify images from registry.redhat.io can be pulled @ 03/09/26 10:31:20.961 "ts"="2026-03-09 10:31:20.961556" "level"=0 "msg"="Starting image pull verification" "namespace"="openshift-nfd" "imageRepository"="registry.redhat.io" "imageName"="ose-node-feature-discovery" "startTime"="2026-03-09T10:31:20Z" "ts"="2026-03-09 10:31:21.063281" "level"=0 "msg"="Container waiting" "pod"="nfd-gc-685d666965-pmgj7" "container"="nfd-gc" "reason"="CreateContainerError" "message"="container create failed: executable file `nfd-gc` not found in $PATH: No such file or directory\n" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "ts"="2026-03-09 10:31:21.063340" "level"=0 "msg"="Successfully pulled image" "pod"="nfd-master-59c77c96f9-mf5fm" "container"="nfd-master" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "imageID"="registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:57a132bdf888ea95345027876c952d09ad72f43a5b4488794bf4f6c45f8520da" "ts"="2026-03-09 10:31:21.063367" "level"=0 "msg"="Successfully pulled image" "pod"="nfd-worker-g5nvf" "container"="nfd-worker" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "imageID"="registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:57a132bdf888ea95345027876c952d09ad72f43a5b4488794bf4f6c45f8520da" "ts"="2026-03-09 10:31:21.063404" "level"=0 "msg"="Successfully pulled image" "pod"="nfd-worker-z4lcj" "container"="nfd-worker" "image"="registry.redhat.io/openshift4/ose-node-feature-discovery:latest" "imageID"="registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:57a132bdf888ea95345027876c952d09ad72f43a5b4488794bf4f6c45f8520da" "ts"="2026-03-09 10:31:21.063414" "level"=0 "msg"="Image pull verification completed successfully" "namespace"="openshift-nfd" "imageRepository"="registry.redhat.io" "imageName"="ose-node-feature-discovery" "duration"="101.862899ms" "endTime"="2026-03-09T10:31:21Z" "ts"="2026-03-09 10:31:21.063598" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:31:21.063711" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-pullsecret-hcp-cluster-pnm7dk" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:31:22.198453" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-pullsecret-hcp-cluster-pnm7dk" "resourceGroup"="" "ts"="2026-03-09 10:31:22.797001" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz" "ts"="2026-03-09 10:31:22.797202" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz" "ts"="2026-03-09 10:31:22.797291" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:31:22.798495" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-111" "ts"="2026-03-09 10:31:22.798537" "level"=0 "msg"="collecting deployments" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:31:23.544608" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-111" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:31:23.994258" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-4w8kdv" "deployment"="pooledNonMsiScopedAssignments-p2h32vyw2l2ma" "ts"="2026-03-09 10:31:24.616274" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-111" "deployment"="msi-deployment-111" "ts"="2026-03-09 10:31:25.197957" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-4w8kdv" "deployment"="customer-infra-pullsecret-hcp-cluster-pnm7dk" "ts"="2026-03-09 10:31:26.294381" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:31:26.294584" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:31:26.294775" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:40:41.310398" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:40:41.310472" "level"=0 "msg"="deleting resource group" "resourceGroup"="pullsecret-test-4w8kdv" "ts"="2026-03-09 10:41:53.715248" "level"=0 "msg"="finished deleting created resources" • [1865.902 seconds] ------------------------------ Ran 1 of 1 Specs in 1865.903 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure rectly returned 409 conflict error while cluster is deploying" "ts"="2026-03-09 10:14:23.458478" "level"=0 "msg"="Cluster provisioning state updated" "provisioningState"="Provisioning" "ts"="2026-03-09 10:19:24.557785" "level"=0 "msg"="Cluster provisioning state updated" "provisioningState"="Succeeded" STEP: creating 3 admin credentials for the ready cluster @ 03/09/26 10:19:24.557 STEP: requesting admin credential 1 @ 03/09/26 10:19:24.557 STEP: validating admin credential 1 works @ 03/09/26 10:20:05.694 "ts"="2026-03-09 10:20:05.972578" "level"=0 "msg"="successfully verified admin credential" "credentialNumber"=1 STEP: requesting admin credential 2 @ 03/09/26 10:20:05.972 STEP: validating admin credential 2 works @ 03/09/26 10:20:57.477 "ts"="2026-03-09 10:20:57.740284" "level"=0 "msg"="successfully verified admin credential" "credentialNumber"=2 STEP: requesting admin credential 3 @ 03/09/26 10:20:57.74 STEP: validating admin credential 3 works @ 03/09/26 10:21:48.967 "ts"="2026-03-09 10:21:49.236550" "level"=0 "msg"="successfully verified admin credential" "credentialNumber"=3 STEP: revoking all cluster admin credentials via ARO HCP RP API @ 03/09/26 10:21:49.236 STEP: validating all admin credentials now fail after revocation @ 03/09/26 10:25:10.781 STEP: verifying admin credential 1 now fails @ 03/09/26 10:25:10.781 "ts"="2026-03-09 10:25:26.066040" "level"=0 "msg"="successfully verified admin credential fails after revocation" "credentialNumber"=1 STEP: verifying admin credential 2 now fails @ 03/09/26 10:25:26.066 "ts"="2026-03-09 10:25:41.333708" "level"=0 "msg"="successfully verified admin credential fails after revocation" "credentialNumber"=2 STEP: verifying admin credential 3 now fails @ 03/09/26 10:25:41.333 "ts"="2026-03-09 10:25:56.602453" "level"=0 "msg"="successfully verified admin credential fails after revocation" "credentialNumber"=3 STEP: verifying new admin credentials can still be requested after revocation @ 03/09/26 10:25:56.602 STEP: verifying new admin credentials work after revocation @ 03/09/26 10:26:18.071 "ts"="2026-03-09 10:26:18.431687" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:18.431817" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-admin-cred-lifecycle-lj6w4s-6zmwvc" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:26:19.414480" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-admin-cred-lifecycle-lj6w4s-6zmwvc" "resourceGroup"="" "ts"="2026-03-09 10:26:20.165546" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-1fc3cb04d84cb6ff56f76d767f6559b3ce3af645ad2734354d7330138cf8842a.yaml.gz" "ts"="2026-03-09 10:26:20.165703" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-1fc3cb04d84cb6ff56f76d767f6559b3ce3af645ad2734354d7330138cf8842a.yaml.gz" "ts"="2026-03-09 10:26:20.165816" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:20.166921" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-270" "ts"="2026-03-09 10:26:20.167041" "level"=0 "msg"="collecting deployments" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:26:20.806873" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-270" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:21.265617" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "deployment"="pooledNonMsiScopedAssignments-k537z423zbcxa" "ts"="2026-03-09 10:26:21.801552" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-270" "deployment"="msi-deployment-270" "ts"="2026-03-09 10:26:22.266037" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "deployment"="customer-infra-admin-cred-lifecycle-lj6w4s-6zmwvc" "ts"="2026-03-09 10:26:23.393023" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:23.393297" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:23.393480" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:36:18.417677" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:36:18.417755" "level"=0 "msg"="deleting resource group" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:37:44.999631" "level"=0 "msg"="finished deleting created resources" • [1617.181 seconds] ------------------------------ Ran 1 of 1 Specs in 1617.181 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') rectly returned 409 conflict error while cluster is deploying" "ts"="2026-03-09 10:14:23.458478" "level"=0 "msg"="Cluster provisioning state updated" "provisioningState"="Provisioning" "ts"="2026-03-09 10:19:24.557785" "level"=0 "msg"="Cluster provisioning state updated" "provisioningState"="Succeeded" STEP: creating 3 admin credentials for the ready cluster @ 03/09/26 10:19:24.557 STEP: requesting admin credential 1 @ 03/09/26 10:19:24.557 STEP: validating admin credential 1 works @ 03/09/26 10:20:05.694 "ts"="2026-03-09 10:20:05.972578" "level"=0 "msg"="successfully verified admin credential" "credentialNumber"=1 STEP: requesting admin credential 2 @ 03/09/26 10:20:05.972 STEP: validating admin credential 2 works @ 03/09/26 10:20:57.477 "ts"="2026-03-09 10:20:57.740284" "level"=0 "msg"="successfully verified admin credential" "credentialNumber"=2 STEP: requesting admin credential 3 @ 03/09/26 10:20:57.74 STEP: validating admin credential 3 works @ 03/09/26 10:21:48.967 "ts"="2026-03-09 10:21:49.236550" "level"=0 "msg"="successfully verified admin credential" "credentialNumber"=3 STEP: revoking all cluster admin credentials via ARO HCP RP API @ 03/09/26 10:21:49.236 STEP: validating all admin credentials now fail after revocation @ 03/09/26 10:25:10.781 STEP: verifying admin credential 1 now fails @ 03/09/26 10:25:10.781 "ts"="2026-03-09 10:25:26.066040" "level"=0 "msg"="successfully verified admin credential fails after revocation" "credentialNumber"=1 STEP: verifying admin credential 2 now fails @ 03/09/26 10:25:26.066 "ts"="2026-03-09 10:25:41.333708" "level"=0 "msg"="successfully verified admin credential fails after revocation" "credentialNumber"=2 STEP: verifying admin credential 3 now fails @ 03/09/26 10:25:41.333 "ts"="2026-03-09 10:25:56.602453" "level"=0 "msg"="successfully verified admin credential fails after revocation" "credentialNumber"=3 STEP: verifying new admin credentials can still be requested after revocation @ 03/09/26 10:25:56.602 STEP: verifying new admin credentials work after revocation @ 03/09/26 10:26:18.071 "ts"="2026-03-09 10:26:18.431687" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:18.431817" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-admin-cred-lifecycle-lj6w4s-6zmwvc" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:26:19.414480" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-admin-cred-lifecycle-lj6w4s-6zmwvc" "resourceGroup"="" "ts"="2026-03-09 10:26:20.165546" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-1fc3cb04d84cb6ff56f76d767f6559b3ce3af645ad2734354d7330138cf8842a.yaml.gz" "ts"="2026-03-09 10:26:20.165703" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-1fc3cb04d84cb6ff56f76d767f6559b3ce3af645ad2734354d7330138cf8842a.yaml.gz" "ts"="2026-03-09 10:26:20.165816" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:20.166921" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-270" "ts"="2026-03-09 10:26:20.167041" "level"=0 "msg"="collecting deployments" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:26:20.806873" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-270" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:21.265617" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "deployment"="pooledNonMsiScopedAssignments-k537z423zbcxa" "ts"="2026-03-09 10:26:21.801552" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-270" "deployment"="msi-deployment-270" "ts"="2026-03-09 10:26:22.266037" "level"=0 "msg"="collecting operations" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "deployment"="customer-infra-admin-cred-lifecycle-lj6w4s-6zmwvc" "ts"="2026-03-09 10:26:23.393023" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:23.393297" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:23.393480" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:36:18.417677" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:36:18.417755" "level"=0 "msg"="deleting resource group" "resourceGroup"="admin-credential-lifecycle-test-sn2hxh" "ts"="2026-03-09 10:37:44.999631" "level"=0 "msg"="finished deleting created resources" • [1617.181 seconds] ------------------------------ Ran 1 of 1 Specs in 1617.181 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to lifecycle and confirm external auth on a cluster [PreLaunchSetup:None, Importance:High, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:43 STEP: creating resource group for the HCP cluster @ 03/09/26 10:10:47.913 "ts"="2026-03-09 10:10:47.914050" "level"=0 "msg"="creating resource group" "resourceGroup"="ea-list-wn5qp9" STEP: creating cluster parameters @ 03/09/26 10:10:52.017 STEP: creating customer resources @ 03/09/26 10:10:52.017 STEP: creating HCP cluster @ 03/09/26 10:11:51.327 "ts"="2026-03-09 10:11:51.328220" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ea-listc25m5z" "resourceGroup"="ea-list-wn5qp9" STEP: create an external auth and confirm it's in a succeeded state @ 03/09/26 10:18:24.966 STEP: confirming we're only allowed to create a single external auth @ 03/09/26 10:18:36.608 STEP: listing all external auth configs to verify a list call works @ 03/09/26 10:18:36.792 STEP: comparing ARM results with expected external auth config @ 03/09/26 10:18:36.889 STEP: updating the external auth to a different prefix and confirming the update works @ 03/09/26 10:18:36.889 "ts"="2026-03-09 10:18:47.514446" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:18:47.514589" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ea-listc25m5z-fjp8qc" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:18:48.351279" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ea-listc25m5z-fjp8qc" "resourceGroup"="" "ts"="2026-03-09 10:18:49.146345" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz" "ts"="2026-03-09 10:18:49.146642" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz" "ts"="2026-03-09 10:18:49.146811" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:18:49.147987" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-203" "ts"="2026-03-09 10:18:49.148105" "level"=0 "msg"="collecting deployments" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:18:49.865857" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-203" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:50.493468" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-wn5qp9" "deployment"="pooledNonMsiScopedAssignments-ay2mmxtcewcro" "ts"="2026-03-09 10:18:50.976669" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-203" "deployment"="msi-deployment-203" "ts"="2026-03-09 10:18:51.629276" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-wn5qp9" "deployment"="customer-infra-ea-listc25m5z-fjp8qc" "ts"="2026-03-09 10:18:52.728537" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:18:52.728722" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:18:52.728916" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:28:57.962897" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:28:57.962967" "level"=0 "msg"="deleting resource group" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:30:24.670320" "level"=0 "msg"="finished deleting created resources" • [1176.795 seconds] ------------------------------ Ran 1 of 1 Specs in 1176.796 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to lifecycle and confirm external auth on a cluster [PreLaunchSetup:None, Importance:High, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:43 STEP: creating resource group for the HCP cluster @ 03/09/26 10:10:47.913 "ts"="2026-03-09 10:10:47.914050" "level"=0 "msg"="creating resource group" "resourceGroup"="ea-list-wn5qp9" STEP: creating cluster parameters @ 03/09/26 10:10:52.017 STEP: creating customer resources @ 03/09/26 10:10:52.017 STEP: creating HCP cluster @ 03/09/26 10:11:51.327 "ts"="2026-03-09 10:11:51.328220" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ea-listc25m5z" "resourceGroup"="ea-list-wn5qp9" STEP: create an external auth and confirm it's in a succeeded state @ 03/09/26 10:18:24.966 STEP: confirming we're only allowed to create a single external auth @ 03/09/26 10:18:36.608 STEP: listing all external auth configs to verify a list call works @ 03/09/26 10:18:36.792 STEP: comparing ARM results with expected external auth config @ 03/09/26 10:18:36.889 STEP: updating the external auth to a different prefix and confirming the update works @ 03/09/26 10:18:36.889 "ts"="2026-03-09 10:18:47.514446" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:18:47.514589" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ea-listc25m5z-fjp8qc" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:18:48.351279" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ea-listc25m5z-fjp8qc" "resourceGroup"="" "ts"="2026-03-09 10:18:49.146345" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz" "ts"="2026-03-09 10:18:49.146642" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz" "ts"="2026-03-09 10:18:49.146811" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:18:49.147987" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-203" "ts"="2026-03-09 10:18:49.148105" "level"=0 "msg"="collecting deployments" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:18:49.865857" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-203" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:50.493468" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-wn5qp9" "deployment"="pooledNonMsiScopedAssignments-ay2mmxtcewcro" "ts"="2026-03-09 10:18:50.976669" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-203" "deployment"="msi-deployment-203" "ts"="2026-03-09 10:18:51.629276" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-wn5qp9" "deployment"="customer-infra-ea-listc25m5z-fjp8qc" "ts"="2026-03-09 10:18:52.728537" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:18:52.728722" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:18:52.728916" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:28:57.962897" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:28:57.962967" "level"=0 "msg"="deleting resource group" "resourceGroup"="ea-list-wn5qp9" "ts"="2026-03-09 10:30:24.670320" "level"=0 "msg"="finished deleting created resources" • [1176.795 seconds] ------------------------------ Ran 1 of 1 Specs in 1176.796 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure r" "error"="failed to create HCP cluster cluster-nsg-reuse: failed starting cluster creation \"cluster-nsg-reuse\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-bpt8vs\": PUT https://rp.j2548992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-bpt8vs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nsg-reuse\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Network Security Group '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-bpt8vs/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name1' is already in use by another cluster\"\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-03-09 10:18:28.290255" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:18:28.290383" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-subnet-reuse-45vbfj" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:18:29.058708" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-cluster-65khst" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:18:29.170588" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-cluster-65khst" "resourceGroup"="" "ts"="2026-03-09 10:18:29.882612" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-subnet-reuse-45vbfj" "resourceGroup"="" "ts"="2026-03-09 10:18:30.638637" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-6950b6d79a3c274e15675866beef4b6fc9a309f9234c1ea3a6f0c13bb6db0b9f.yaml.gz" "ts"="2026-03-09 10:18:30.638904" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-6950b6d79a3c274e15675866beef4b6fc9a309f9234c1ea3a6f0c13bb6db0b9f.yaml.gz" "ts"="2026-03-09 10:18:30.639032" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:18:30.640712" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "ts"="2026-03-09 10:18:30.640723" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:18:30.640766" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "ts"="2026-03-09 10:18:31.410301" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:32.033938" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:32.679565" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="pooledNonMsiScopedAssignments-67tmrjj4do2cc" "ts"="2026-03-09 10:18:33.275648" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="msi-deployment-149" "ts"="2026-03-09 10:18:33.763144" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="msi-deployment-98" "ts"="2026-03-09 10:18:34.410448" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="pooledNonMsiScopedAssignments-j3gkygvtvkvjs" "ts"="2026-03-09 10:18:36.209875" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="customer-infra-basic-cluster-65khst" "ts"="2026-03-09 10:18:36.959561" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="customer-infra-cluster-subnet-reuse-45vbfj" "ts"="2026-03-09 10:18:37.924952" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:18:37.925145" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:18:37.925329" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:28:02.381628" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:28:02.381678" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:29:36.898827" "level"=0 "msg"="finished deleting created resources" • [1129.013 seconds] ------------------------------ Ran 1 of 1 Specs in 1129.013 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') r" "error"="failed to create HCP cluster cluster-nsg-reuse: failed starting cluster creation \"cluster-nsg-reuse\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-bpt8vs\": PUT https://rp.j2548992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-bpt8vs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nsg-reuse\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Network Security Group '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-bpt8vs/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name1' is already in use by another cluster\"\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-03-09 10:18:28.290255" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:18:28.290383" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-subnet-reuse-45vbfj" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:18:29.058708" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-cluster-65khst" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:18:29.170588" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-cluster-65khst" "resourceGroup"="" "ts"="2026-03-09 10:18:29.882612" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-subnet-reuse-45vbfj" "resourceGroup"="" "ts"="2026-03-09 10:18:30.638637" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-6950b6d79a3c274e15675866beef4b6fc9a309f9234c1ea3a6f0c13bb6db0b9f.yaml.gz" "ts"="2026-03-09 10:18:30.638904" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-6950b6d79a3c274e15675866beef4b6fc9a309f9234c1ea3a6f0c13bb6db0b9f.yaml.gz" "ts"="2026-03-09 10:18:30.639032" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:18:30.640712" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "ts"="2026-03-09 10:18:30.640723" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:18:30.640766" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "ts"="2026-03-09 10:18:31.410301" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:32.033938" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:18:32.679565" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="pooledNonMsiScopedAssignments-67tmrjj4do2cc" "ts"="2026-03-09 10:18:33.275648" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-149" "deployment"="msi-deployment-149" "ts"="2026-03-09 10:18:33.763144" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-98" "deployment"="msi-deployment-98" "ts"="2026-03-09 10:18:34.410448" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="pooledNonMsiScopedAssignments-j3gkygvtvkvjs" "ts"="2026-03-09 10:18:36.209875" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="customer-infra-basic-cluster-65khst" "ts"="2026-03-09 10:18:36.959561" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "deployment"="customer-infra-cluster-subnet-reuse-45vbfj" "ts"="2026-03-09 10:18:37.924952" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:18:37.925145" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:18:37.925329" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:28:02.381628" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:28:02.381678" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-cluster-nsg-subnet-reuse-bpt8vs" "ts"="2026-03-09 10:29:36.898827" "level"=0 "msg"="finished deleting created resources" • [1129.013 seconds] ------------------------------ Ran 1 of 1 Specs in 1129.013 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster with Image Registry not present [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:37 STEP: creating a resource group @ 03/09/26 10:10:47.846 "ts"="2026-03-09 10:10:47.846846" "level"=0 "msg"="creating resource group" "resourceGroup"="disabled-image-registry-g9p7wd" STEP: creating cluster parameters @ 03/09/26 10:10:51.715 STEP: creating customer resources @ 03/09/26 10:10:51.715 STEP: creating the hcp cluster with the image registry disabled @ 03/09/26 10:11:40.481 "ts"="2026-03-09 10:11:40.481447" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="disabled-image-registry-hcp-cluster" "resourceGroup"="disabled-image-registry-g9p7wd" STEP: getting credentials @ 03/09/26 10:19:25.198 "ts"="2026-03-09 10:20:08.134282" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:20:08.134408" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-disabled-image-registry-hcp-cluster-nr6j4l" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:20:09.348970" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-disabled-image-registry-hcp-cluster-nr6j4l" "resourceGroup"="" "ts"="2026-03-09 10:20:10.522949" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-f66366cb92003bba305ea6fa81a0011f709bf02a764a23c42e292ad6f9754eee.yaml.gz" "ts"="2026-03-09 10:20:10.523069" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-f66366cb92003bba305ea6fa81a0011f709bf02a764a23c42e292ad6f9754eee.yaml.gz" "ts"="2026-03-09 10:20:10.523157" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:20:10.524439" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "ts"="2026-03-09 10:20:10.524485" "level"=0 "msg"="collecting deployments" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:20:11.188104" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:11.573893" "level"=0 "msg"="collecting operations" "resourceGroup"="disabled-image-registry-g9p7wd" "deployment"="pooledNonMsiScopedAssignments-qmh74zogaodhg" "ts"="2026-03-09 10:20:12.302658" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="msi-deployment-7" "ts"="2026-03-09 10:20:12.550050" "level"=0 "msg"="collecting operations" "resourceGroup"="disabled-image-registry-g9p7wd" "deployment"="customer-infra-disabled-image-registry-hcp-cluster-nr6j4l" "ts"="2026-03-09 10:20:13.526428" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:20:13.526614" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:20:13.526761" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:28:58.626840" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:28:58.626992" "level"=0 "msg"="deleting resource group" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:30:28.961769" "level"=0 "msg"="finished deleting created resources" • [1181.122 seconds] ------------------------------ Ran 1 of 1 Specs in 1181.122 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster with Image Registry not present [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:37 STEP: creating a resource group @ 03/09/26 10:10:47.846 "ts"="2026-03-09 10:10:47.846846" "level"=0 "msg"="creating resource group" "resourceGroup"="disabled-image-registry-g9p7wd" STEP: creating cluster parameters @ 03/09/26 10:10:51.715 STEP: creating customer resources @ 03/09/26 10:10:51.715 STEP: creating the hcp cluster with the image registry disabled @ 03/09/26 10:11:40.481 "ts"="2026-03-09 10:11:40.481447" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="disabled-image-registry-hcp-cluster" "resourceGroup"="disabled-image-registry-g9p7wd" STEP: getting credentials @ 03/09/26 10:19:25.198 "ts"="2026-03-09 10:20:08.134282" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:20:08.134408" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-disabled-image-registry-hcp-cluster-nr6j4l" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:20:09.348970" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-disabled-image-registry-hcp-cluster-nr6j4l" "resourceGroup"="" "ts"="2026-03-09 10:20:10.522949" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-f66366cb92003bba305ea6fa81a0011f709bf02a764a23c42e292ad6f9754eee.yaml.gz" "ts"="2026-03-09 10:20:10.523069" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-f66366cb92003bba305ea6fa81a0011f709bf02a764a23c42e292ad6f9754eee.yaml.gz" "ts"="2026-03-09 10:20:10.523157" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:20:10.524439" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "ts"="2026-03-09 10:20:10.524485" "level"=0 "msg"="collecting deployments" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:20:11.188104" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:11.573893" "level"=0 "msg"="collecting operations" "resourceGroup"="disabled-image-registry-g9p7wd" "deployment"="pooledNonMsiScopedAssignments-qmh74zogaodhg" "ts"="2026-03-09 10:20:12.302658" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="msi-deployment-7" "ts"="2026-03-09 10:20:12.550050" "level"=0 "msg"="collecting operations" "resourceGroup"="disabled-image-registry-g9p7wd" "deployment"="customer-infra-disabled-image-registry-hcp-cluster-nr6j4l" "ts"="2026-03-09 10:20:13.526428" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:20:13.526614" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:20:13.526761" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:28:58.626840" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:28:58.626992" "level"=0 "msg"="deleting resource group" "resourceGroup"="disabled-image-registry-g9p7wd" "ts"="2026-03-09 10:30:28.961769" "level"=0 "msg"="finished deleting created resources" • [1181.122 seconds] ------------------------------ Ran 1 of 1 Specs in 1181.122 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to update nodepool replicas and autoscaling [PreLaunchSetup:None, Importance:High, Positivity:Positive, ARO-HCP-RP-API-Compatible, Speed:Slow] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35 STEP: creating a resource group @ 03/09/26 10:30:58.632 "ts"="2026-03-09 10:30:58.632647" "level"=0 "msg"="creating resource group" "resourceGroup"="nodepool-update-nodes-2hkwzb" STEP: creating cluster parameters @ 03/09/26 10:31:01.566 STEP: creating customer resources @ 03/09/26 10:31:01.566 STEP: creating the HCP cluster @ 03/09/26 10:31:59.488 "ts"="2026-03-09 10:31:59.488288" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-update-nodes-hcp-cluster" "resourceGroup"="nodepool-update-nodes-2hkwzb" STEP: getting admin credentials for the cluster @ 03/09/26 10:38:53.171 STEP: creating the node pools in parallel @ 03/09/26 10:39:34.897 STEP: verifying nodes count and status after initial creation @ 03/09/26 10:48:48.949 STEP: scaling up the nodepool replicas from 2 to 3 replicas @ 03/09/26 10:48:49.365 STEP: verifying nodes count and status after scaling up @ 03/09/26 10:55:42.402 STEP: scaling down the nodepool replicas from 3 to 2 replicas @ 03/09/26 10:55:42.864 STEP: verifying nodes count and status after scaling down @ 03/09/26 11:00:25.346 STEP: updating the one-replica nodepool replicas to 0 and enabling autoscaling with a PATCH @ 03/09/26 11:00:25.791 STEP: verifying nodes count and status after enabling autoscaling @ 03/09/26 11:06:59.517 "ts"="2026-03-09 11:06:59.944387" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 11:06:59.944778" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-update-nodes-hcp-cluster-b8v7wz" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:07:01.436736" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-update-nodes-hcp-cluster-b8v7wz" "resourceGroup"="" "ts"="2026-03-09 11:07:02.564999" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz" "ts"="2026-03-09 11:07:02.565206" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz" "ts"="2026-03-09 11:07:02.565440" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 11:07:02.568263" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "ts"="2026-03-09 11:07:02.568360" "level"=0 "msg"="collecting deployments" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:07:03.417030" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 11:07:03.810298" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-2hkwzb" "deployment"="pooledNonMsiScopedAssignments-w3k44uxsb3qey" "ts"="2026-03-09 11:07:04.354472" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="msi-deployment-7" "ts"="2026-03-09 11:07:04.842753" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-2hkwzb" "deployment"="customer-infra-np-update-nodes-hcp-cluster-b8v7wz" "ts"="2026-03-09 11:07:06.013930" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 11:07:06.015009" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 11:07:06.015446" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:16:30.916136" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:16:30.916215" "level"=0 "msg"="deleting resource group" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:17:42.284327" "level"=0 "msg"="finished deleting created resources" • [4014.371 seconds] ------------------------------ Ran 1 of 1 Specs in 4014.372 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to update nodepool replicas and autoscaling [PreLaunchSetup:None, Importance:High, Positivity:Positive, ARO-HCP-RP-API-Compatible, Speed:Slow] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35 STEP: creating a resource group @ 03/09/26 10:30:58.632 "ts"="2026-03-09 10:30:58.632647" "level"=0 "msg"="creating resource group" "resourceGroup"="nodepool-update-nodes-2hkwzb" STEP: creating cluster parameters @ 03/09/26 10:31:01.566 STEP: creating customer resources @ 03/09/26 10:31:01.566 STEP: creating the HCP cluster @ 03/09/26 10:31:59.488 "ts"="2026-03-09 10:31:59.488288" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-update-nodes-hcp-cluster" "resourceGroup"="nodepool-update-nodes-2hkwzb" STEP: getting admin credentials for the cluster @ 03/09/26 10:38:53.171 STEP: creating the node pools in parallel @ 03/09/26 10:39:34.897 STEP: verifying nodes count and status after initial creation @ 03/09/26 10:48:48.949 STEP: scaling up the nodepool replicas from 2 to 3 replicas @ 03/09/26 10:48:49.365 STEP: verifying nodes count and status after scaling up @ 03/09/26 10:55:42.402 STEP: scaling down the nodepool replicas from 3 to 2 replicas @ 03/09/26 10:55:42.864 STEP: verifying nodes count and status after scaling down @ 03/09/26 11:00:25.346 STEP: updating the one-replica nodepool replicas to 0 and enabling autoscaling with a PATCH @ 03/09/26 11:00:25.791 STEP: verifying nodes count and status after enabling autoscaling @ 03/09/26 11:06:59.517 "ts"="2026-03-09 11:06:59.944387" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 11:06:59.944778" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-update-nodes-hcp-cluster-b8v7wz" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:07:01.436736" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-update-nodes-hcp-cluster-b8v7wz" "resourceGroup"="" "ts"="2026-03-09 11:07:02.564999" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz" "ts"="2026-03-09 11:07:02.565206" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz" "ts"="2026-03-09 11:07:02.565440" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 11:07:02.568263" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "ts"="2026-03-09 11:07:02.568360" "level"=0 "msg"="collecting deployments" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:07:03.417030" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 11:07:03.810298" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-2hkwzb" "deployment"="pooledNonMsiScopedAssignments-w3k44uxsb3qey" "ts"="2026-03-09 11:07:04.354472" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-7" "deployment"="msi-deployment-7" "ts"="2026-03-09 11:07:04.842753" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-2hkwzb" "deployment"="customer-infra-np-update-nodes-hcp-cluster-b8v7wz" "ts"="2026-03-09 11:07:06.013930" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 11:07:06.015009" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 11:07:06.015446" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:16:30.916136" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:16:30.916215" "level"=0 "msg"="deleting resource group" "resourceGroup"="nodepool-update-nodes-2hkwzb" "ts"="2026-03-09 11:17:42.284327" "level"=0 "msg"="finished deleting created resources" • [4014.371 seconds] ------------------------------ Ran 1 of 1 Specs in 4014.372 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster with custom autoscaling [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:36 STEP: creating a resource group @ 03/09/26 10:10:47.871 "ts"="2026-03-09 10:10:47.871225" "level"=0 "msg"="creating resource group" "resourceGroup"="autoscaling-cluster-sk2ph2" STEP: creating customer resources @ 03/09/26 10:10:52.131 STEP: creating the cluster @ 03/09/26 10:11:49.528 "ts"="2026-03-09 10:11:49.528704" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="autoscaling-hcp-cluster" "resourceGroup"="autoscaling-cluster-sk2ph2" STEP: ensuring the custom autoscaling was honored @ 03/09/26 10:18:23.022 STEP: creating the node pool @ 03/09/26 10:18:23.123 STEP: patching the cluster to set maxNodesTotal @ 03/09/26 10:25:36.123 "ts"="2026-03-09 10:26:06.967601" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:06.967744" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-autoscaling-hcp-cluster-rsq687" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:26:07.974257" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-autoscaling-hcp-cluster-rsq687" "resourceGroup"="" "ts"="2026-03-09 10:26:08.667626" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz" "ts"="2026-03-09 10:26:08.667748" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz" "ts"="2026-03-09 10:26:08.667828" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:08.668858" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-120" "ts"="2026-03-09 10:26:08.668902" "level"=0 "msg"="collecting deployments" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:26:09.302296" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-120" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:09.765368" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-sk2ph2" "deployment"="pooledNonMsiScopedAssignments-e4ixwyxsfjnje" "ts"="2026-03-09 10:26:10.282160" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-120" "deployment"="msi-deployment-120" "ts"="2026-03-09 10:26:10.729734" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-sk2ph2" "deployment"="customer-infra-autoscaling-hcp-cluster-rsq687" "ts"="2026-03-09 10:26:11.723104" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:11.723310" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:11.723489" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:36:56.784054" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:36:56.784111" "level"=0 "msg"="deleting resource group" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:38:22.700123" "level"=0 "msg"="finished deleting created resources" • [1654.878 seconds] ------------------------------ Ran 1 of 1 Specs in 1654.879 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create an HCP cluster with custom autoscaling [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:36 STEP: creating a resource group @ 03/09/26 10:10:47.871 "ts"="2026-03-09 10:10:47.871225" "level"=0 "msg"="creating resource group" "resourceGroup"="autoscaling-cluster-sk2ph2" STEP: creating customer resources @ 03/09/26 10:10:52.131 STEP: creating the cluster @ 03/09/26 10:11:49.528 "ts"="2026-03-09 10:11:49.528704" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="autoscaling-hcp-cluster" "resourceGroup"="autoscaling-cluster-sk2ph2" STEP: ensuring the custom autoscaling was honored @ 03/09/26 10:18:23.022 STEP: creating the node pool @ 03/09/26 10:18:23.123 STEP: patching the cluster to set maxNodesTotal @ 03/09/26 10:25:36.123 "ts"="2026-03-09 10:26:06.967601" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:26:06.967744" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-autoscaling-hcp-cluster-rsq687" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:26:07.974257" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-autoscaling-hcp-cluster-rsq687" "resourceGroup"="" "ts"="2026-03-09 10:26:08.667626" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz" "ts"="2026-03-09 10:26:08.667748" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz" "ts"="2026-03-09 10:26:08.667828" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:26:08.668858" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-120" "ts"="2026-03-09 10:26:08.668902" "level"=0 "msg"="collecting deployments" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:26:09.302296" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-120" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:26:09.765368" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-sk2ph2" "deployment"="pooledNonMsiScopedAssignments-e4ixwyxsfjnje" "ts"="2026-03-09 10:26:10.282160" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-120" "deployment"="msi-deployment-120" "ts"="2026-03-09 10:26:10.729734" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-sk2ph2" "deployment"="customer-infra-autoscaling-hcp-cluster-rsq687" "ts"="2026-03-09 10:26:11.723104" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:26:11.723310" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:26:11.723489" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:36:56.784054" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:36:56.784111" "level"=0 "msg"="deleting resource group" "resourceGroup"="autoscaling-cluster-sk2ph2" "ts"="2026-03-09 10:38:22.700123" "level"=0 "msg"="finished deleting created resources" • [1654.878 seconds] ------------------------------ Ran 1 of 1 Specs in 1654.879 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure -connectivity-test" "resourceGroup"="e2e-cidr-connectivity-9m625b" STEP: getting cluster details @ 03/09/26 10:19:27.349 STEP: verifying authorized CIDRs contains VM IP @ 03/09/26 10:19:27.456 STEP: testing connectivity from authorized VM @ 03/09/26 10:19:27.456 STEP: VM received HTTP status code: 200 @ 03/09/26 10:19:59.766 STEP: testing connectivity from current machine (should be blocked) @ 03/09/26 10:19:59.766 STEP: verifying VM can access cluster API with credentials @ 03/09/26 10:19:59.923 STEP: kubectl command executed from authorized VM @ 03/09/26 10:21:22.297 STEP: Successfully retrieved node list with 0 items @ 03/09/26 10:21:22.298 STEP: verifying aggregated API services from authorized VM @ 03/09/26 10:21:22.298 STEP: creating the node pool @ 03/09/26 10:21:53.737 STEP: creating an app registration with a client secret @ 03/09/26 10:30:07.403 "ts"="2026-03-09 10:30:07.403848" "level"=0 "msg"="creating app registration" "appName"="aro-hcp-e2e-5476705466293221631" STEP: creating an external auth config with a prefix @ 03/09/26 10:30:10.775 STEP: verifying ExternalAuth is in a Succeeded state @ 03/09/26 10:30:21.402 STEP: creating a cluster role binding for the entra application via VM @ 03/09/26 10:30:21.484 STEP: creating a rest config using OIDC authentication @ 03/09/26 10:30:52.984 STEP: Successfully retrieved 2 nodes using external auth @ 03/09/26 10:33:07.671 STEP: creating the console OAuth client secret for external auth via VM @ 03/09/26 10:33:07.671 STEP: verifying all cluster operators are healthy from authorized VM @ 03/09/26 10:33:39.175 STEP: updating cluster to remove VM from authorized CIDRs @ 03/09/26 10:34:10.768 STEP: verifying VM is now blocked from API access @ 03/09/26 10:35:41.728 STEP: Connection correctly blocked after removing VM from authorized CIDRs @ 03/09/26 10:36:13.066 "ts"="2026-03-09 10:36:13.066897" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:36:13.067042" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cidr-connectivity-test-pqrp5j" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:36:14.536029" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cidr-connectivity-test-pqrp5j" "resourceGroup"="" "ts"="2026-03-09 10:36:15.233877" "level"=0 "msg"="Dumping deployment operations." "deployment"="test-vm" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:36:15.351920" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz" "ts"="2026-03-09 10:36:15.352077" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz" "ts"="2026-03-09 10:36:15.352214" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:36:15.354156" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-193" "ts"="2026-03-09 10:36:15.354205" "level"=0 "msg"="collecting deployments" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:36:16.041965" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-193" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:36:16.625752" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-9m625b" "deployment"="test-vm" "ts"="2026-03-09 10:36:17.200292" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-193" "deployment"="msi-deployment-193" "ts"="2026-03-09 10:36:17.846350" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-9m625b" "deployment"="pooledNonMsiScopedAssignments-wihoigd7cpguy" "ts"="2026-03-09 10:36:19.004449" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-9m625b" "deployment"="customer-infra-cidr-connectivity-test-pqrp5j" "ts"="2026-03-09 10:36:19.724799" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:36:19.725051" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:36:19.725300" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:45:14.365122" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:45:14.365180" "level"=0 "msg"="deleting resource group" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:48:29.310761" "level"=0 "msg"="finished deleting created resources" • [2261.432 seconds] ------------------------------ Ran 1 of 1 Specs in 2261.433 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') -connectivity-test" "resourceGroup"="e2e-cidr-connectivity-9m625b" STEP: getting cluster details @ 03/09/26 10:19:27.349 STEP: verifying authorized CIDRs contains VM IP @ 03/09/26 10:19:27.456 STEP: testing connectivity from authorized VM @ 03/09/26 10:19:27.456 STEP: VM received HTTP status code: 200 @ 03/09/26 10:19:59.766 STEP: testing connectivity from current machine (should be blocked) @ 03/09/26 10:19:59.766 STEP: verifying VM can access cluster API with credentials @ 03/09/26 10:19:59.923 STEP: kubectl command executed from authorized VM @ 03/09/26 10:21:22.297 STEP: Successfully retrieved node list with 0 items @ 03/09/26 10:21:22.298 STEP: verifying aggregated API services from authorized VM @ 03/09/26 10:21:22.298 STEP: creating the node pool @ 03/09/26 10:21:53.737 STEP: creating an app registration with a client secret @ 03/09/26 10:30:07.403 "ts"="2026-03-09 10:30:07.403848" "level"=0 "msg"="creating app registration" "appName"="aro-hcp-e2e-5476705466293221631" STEP: creating an external auth config with a prefix @ 03/09/26 10:30:10.775 STEP: verifying ExternalAuth is in a Succeeded state @ 03/09/26 10:30:21.402 STEP: creating a cluster role binding for the entra application via VM @ 03/09/26 10:30:21.484 STEP: creating a rest config using OIDC authentication @ 03/09/26 10:30:52.984 STEP: Successfully retrieved 2 nodes using external auth @ 03/09/26 10:33:07.671 STEP: creating the console OAuth client secret for external auth via VM @ 03/09/26 10:33:07.671 STEP: verifying all cluster operators are healthy from authorized VM @ 03/09/26 10:33:39.175 STEP: updating cluster to remove VM from authorized CIDRs @ 03/09/26 10:34:10.768 STEP: verifying VM is now blocked from API access @ 03/09/26 10:35:41.728 STEP: Connection correctly blocked after removing VM from authorized CIDRs @ 03/09/26 10:36:13.066 "ts"="2026-03-09 10:36:13.066897" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:36:13.067042" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cidr-connectivity-test-pqrp5j" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:36:14.536029" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cidr-connectivity-test-pqrp5j" "resourceGroup"="" "ts"="2026-03-09 10:36:15.233877" "level"=0 "msg"="Dumping deployment operations." "deployment"="test-vm" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:36:15.351920" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz" "ts"="2026-03-09 10:36:15.352077" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz" "ts"="2026-03-09 10:36:15.352214" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:36:15.354156" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-193" "ts"="2026-03-09 10:36:15.354205" "level"=0 "msg"="collecting deployments" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:36:16.041965" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-193" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:36:16.625752" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-9m625b" "deployment"="test-vm" "ts"="2026-03-09 10:36:17.200292" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-193" "deployment"="msi-deployment-193" "ts"="2026-03-09 10:36:17.846350" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-9m625b" "deployment"="pooledNonMsiScopedAssignments-wihoigd7cpguy" "ts"="2026-03-09 10:36:19.004449" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-9m625b" "deployment"="customer-infra-cidr-connectivity-test-pqrp5j" "ts"="2026-03-09 10:36:19.724799" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:36:19.725051" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:36:19.725300" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:45:14.365122" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:45:14.365180" "level"=0 "msg"="deleting resource group" "resourceGroup"="e2e-cidr-connectivity-9m625b" "ts"="2026-03-09 10:48:29.310761" "level"=0 "msg"="finished deleting created resources" • [2261.432 seconds] ------------------------------ Ran 1 of 1 Specs in 2261.433 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Deserializaion Error: invalid character 'i' in literal true (expecting 'r') 09:44:59 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should respect cluster-wide node limits with nodepool autoscaling [PreLaunchSetup:None, Importance:Medium, Positivity:Negative, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122 STEP: creating a resource group @ 03/09/26 10:10:47.883 "ts"="2026-03-09 10:10:47.883143" "level"=0 "msg"="creating resource group" "resourceGroup"="node-limit-test-rmm4pp" STEP: creating cluster with MaxNodesTotal limit @ 03/09/26 10:10:52.983 "ts"="2026-03-09 10:11:49.750674" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="node-limit-cluster" "resourceGroup"="node-limit-test-rmm4pp" STEP: attempting to create nodepool with Min > MaxNodesTotal @ 03/09/26 10:20:15.181 "ts"="2026-03-09 10:20:15.312648" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:20:15.312768" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-node-limit-cluster-nswvmq" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:20:16.325689" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-node-limit-cluster-nswvmq" "resourceGroup"="" "ts"="2026-03-09 10:20:17.593532" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz" "ts"="2026-03-09 10:20:17.593647" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz" "ts"="2026-03-09 10:20:17.593734" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:20:17.594708" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-130" "ts"="2026-03-09 10:20:17.594745" "level"=0 "msg"="collecting deployments" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:20:18.244092" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-130" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:18.683805" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-rmm4pp" "deployment"="pooledNonMsiScopedAssignments-j6nw3ohx7bzo4" "ts"="2026-03-09 10:20:19.205277" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-130" "deployment"="msi-deployment-130" "ts"="2026-03-09 10:20:19.637290" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-rmm4pp" "deployment"="customer-infra-node-limit-cluster-nswvmq" "ts"="2026-03-09 10:20:20.677395" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:20:20.677659" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:20:20.677981" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:30:05.795084" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:30:05.795256" "level"=0 "msg"="deleting resource group" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:31:17.926615" "level"=0 "msg"="finished deleting created resources" • [1230.077 seconds] ------------------------------ Ran 1 of 1 Specs in 1230.077 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r') Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1773051047 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should respect cluster-wide node limits with nodepool autoscaling [PreLaunchSetup:None, Importance:Medium, Positivity:Negative, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122 STEP: creating a resource group @ 03/09/26 10:10:47.883 "ts"="2026-03-09 10:10:47.883143" "level"=0 "msg"="creating resource group" "resourceGroup"="node-limit-test-rmm4pp" STEP: creating cluster with MaxNodesTotal limit @ 03/09/26 10:10:52.983 "ts"="2026-03-09 10:11:49.750674" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="node-limit-cluster" "resourceGroup"="node-limit-test-rmm4pp" STEP: attempting to create nodepool with Min > MaxNodesTotal @ 03/09/26 10:20:15.181 "ts"="2026-03-09 10:20:15.312648" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-09 10:20:15.312768" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-node-limit-cluster-nswvmq" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:20:16.325689" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-node-limit-cluster-nswvmq" "resourceGroup"="" "ts"="2026-03-09 10:20:17.593532" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz" "ts"="2026-03-09 10:20:17.593647" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz" "ts"="2026-03-09 10:20:17.593734" "level"=0 "msg"="collecting debug info" "ts"="2026-03-09 10:20:17.594708" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-130" "ts"="2026-03-09 10:20:17.594745" "level"=0 "msg"="collecting deployments" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:20:18.244092" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-130" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-09 10:20:18.683805" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-rmm4pp" "deployment"="pooledNonMsiScopedAssignments-j6nw3ohx7bzo4" "ts"="2026-03-09 10:20:19.205277" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-130" "deployment"="msi-deployment-130" "ts"="2026-03-09 10:20:19.637290" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-rmm4pp" "deployment"="customer-infra-node-limit-cluster-nswvmq" "ts"="2026-03-09 10:20:20.677395" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-09 10:20:20.677659" "level"=0 "msg"="deleting created resources" "ts"="2026-03-09 10:20:20.677981" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:30:05.795084" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:30:05.795256" "level"=0 "msg"="deleting resource group" "resourceGroup"="node-limit-test-rmm4pp" "ts"="2026-03-09 10:31:17.926615" "level"=0 "msg"="finished deleting created resources" • [1230.077 seconds] ------------------------------ Ran 1 of 1 Specs in 1230.077 seconds SUCCESS! -- 1 Passed \| 0 Failed \| 0 Pending \| 0 Skipped Command Error: <nil> Deserializaion Error: invalid character 'i' in literal true (expecting 'r')
09:20:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2e6c304	n/a	n/a	passed	#4364 (open)	0	n/a
09:04:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head f758798 · merge 523a271	n/a	provision	failed	#4363 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 09:04:07 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-03-09T09:33:46.213Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-09T09:33:48.541Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-03-09T09:33:49.543Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=3ca8d38bbb29123457df59c9bb94bbac5baa3103c704b582a1758d093cbef96d time=2026-03-09T09:37:20.479Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9254400/providers/Microsoft.Resources/deployments/3ca8d38bbb29123457df59c9bb94bbac5baa3103c704b582a1758d093cbef96d/operationStatuses/08584285580565790414\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9254400/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9254400/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j9254400\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 2954c5b2-21d3-4d00-93b2-68482b979c33:3/9/2026 9:35:03 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
08:43:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a72618f · merge a72618f	n/a	n/a	passed	#4362 (merged)	0	n/a
08:28:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8826d13 · merge 8826d13	n/a	n/a	passed	#4357 (merged)	0	n/a
08:23:37	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 100f45d · merge 131806c	n/a	e2e	failed	#4351 (merged)	1	e2e (1) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 08:23:37 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:229]: Unexpected error: <errors.joinError \| 0xc0010fa300>: cluster listing in non-existent resource group: expected error but none occurred node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-97522w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <errors.errorString \| 0xc000da9680>{ s: "cluster listing in non-existent resource group: expected error but none occurred", }, <errors.errorString \| 0xc001034ca0>{ s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-97522w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, <errors.errorString \| 0xc000e6b220>{ s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:229]: Unexpected error: <errors.joinError \| 0xc0010fa300>: cluster listing in non-existent resource group: expected error but none occurred node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-97522w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <errors.errorString \| 0xc000da9680>{ s: "cluster listing in non-existent resource group: expected error but none occurred", }, <errors.errorString \| 0xc001034ca0>{ s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-97522w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, <errors.errorString \| 0xc000e6b220>{ s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6269824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", }, ], } occurred
07:57:32	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head af664ab · merge ad7003e	n/a	n/a	passed	#4188 (merged)	0	n/a
07:12:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e0a41cb · merge 5b9af08	n/a	n/a	passed	#4361 (merged)	0	n/a
06:34:29	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8826d13 · merge 8826d13	n/a	n/a	passed	#4357 (merged)	0	n/a
05:07:25	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 55a8598 · merge 55a8598	n/a	n/a	passed	#4360 (merged)	0	n/a
03:06:25	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 249dc93 · merge 249dc93	n/a	n/a	passed	#4359 (merged)	0	n/a
00:23:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 56807a7 · merge 8826d13	n/a	provision	failed	#4357 (merged)	1	provision (1) error running Shell Step, failed to execute shell command: + set -euo pipefail + ISTIO_NAMESPACE=aks-istio-system + echo '******** Check istio is up and running **********' ****** Check istio is up and running ********** ++ kubectl get pod 00:23:55 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade · suite step graph Show raw failure time=2026-03-09T01:19:23.519Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: Shell\n Command: ./istio.sh\n" time=2026-03-09T01:19:40.786Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="error running Shell Step, failed to execute shell command: + set -euo pipefail\n+ ISTIO_NAMESPACE=aks-istio-system\n+ echo '****** Check istio is up and running **********'\n****** Check istio is up and running **********\n++ kubectl get pods -n aks-istio-system -l istio.io/rev=asm-1-26 --field-selector=status.phase=Running --no-headers\n++ wc -l\n+ ISTIO_PODS_COUNT=2\n+ [[ 2 -lt 2 ]]\n+ echo '****** Download istioctl **********'\n****** Download istioctl ************\n++ uname\n+ OS=Linux\n+ '[' Linux = Darwin ']'\n+ OSEXT=linux\n++ uname -m\n+ LOCAL_ARCH=x86_64\n+ case \"${LOCAL_ARCH}\" in\n+ ISTIO_ARCH=amd64\n+ ISTIO_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz\n+ SHA256_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz -o istio-1.28.3-linux-amd64.tar.gz\n exit status 6"