No runs match your search.
Environment: DEV
| Time (UTC) | Runtime | Job | Region | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|---|---|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4589 (closed) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>. Show raw failuretime=2026-03-24T00:31:24.809Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-24T00:31:25.578Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-24T00:31:26.601Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=7a15931e1f96af913f2cb12fb2d11d6e14f7d5aaaf1b1b54309a877efbe4f835
time=2026-03-24T00:31:56.909Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8096384-mgmt-1/providers/Microsoft.Resources/deployments/7a15931e1f96af913f2cb12fb2d11d6e14f7d5aaaf1b1b54309a877efbe4f835/operationStatuses/08584272945994994062\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is c2458f5d40a058a9b4d76c1a122ba64b.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4586 (merged) | 1 | provision (1)failed pre-install: Unauthorized Show raw failuretime=2026-03-23T23:55:28.903Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-23T23:55:29.544Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b7c83e5834c5e97ef17af96ddbcba9ed7062371a1db1a46171ff7ae1ae137aec] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b7bd3a6bf3160b82af89c345c58b1fd3a936801c8c65fef7ed2111dc71c766a3]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:bef5c287-bf4c-42fe-8749-19e46529efe9]] prometheusSpec:map[externalLabels:map[cluster:prow-j4595712-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j4595712-mgmt-1-lfli.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-f441ffb297654dd29a83464cc6a20f65/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:eb5dcb5c4f6e229d6dfe17b492291e06bea7ba19f080d48a7afc663e4420386c] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j4595712-mgmt-1-lfli.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-3c038c95ecfb4de8b8a9349e9256b31f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-23T23:55:29.544Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T23:55:29.855Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-23T23:55:30.215Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-23T23:55:30.215Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T23:55:30.392Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T23:55:56.499Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed pre-install: Unauthorized" time=2026-03-23T23:55:56.499Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T23:55:56.499Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T23:56:03.840Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed pre-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b7c83e5834c5e97ef17af96ddbcba9ed7062371a1db1a46171ff7ae1ae137aec] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b7bd3a6bf3160b82af89c345c58b1fd3a936801c8c65fef7ed2111dc71c766a3]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:bef5c287-bf4c-42fe-8749-19e46529efe9]] prometheusSpec:map[externalLabels:map[cluster:prow-j4595712-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j4595712-mgmt-1-lfli.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-f441ffb297654dd29a83464cc6a20f65/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:eb5dcb5c4f6e229d6dfe17b492291e06bea7ba19f080d48a7afc663e4420386c] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j4595712-mgmt-1-lfli.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-3c038c95ecfb4de8b8a9349e9256b31f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-23T23:56:03.840Z level=INFO msg="No resource info available in release, skipping resource diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus infoNil=false resourceCount=0 time=2026-03-23T23:56:03.840Z level=INFO msg="Kusto deep link for namespace-level troubleshooting" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus kustoLinkNamespace="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/3yOvaqDQBSEe5/isJWCiuyiheAb3PJWNpd1HdRL9ofdEyTgwwdTmFTpho8Z5jPesd4c4o9fUnbQviKCeLNIrG2gCbwDjvJZM06cC9nIrmpUJdWvVH3b9qoZRUF1TV86bTeKorgOnLZIQRv8nYmGgUSI3oJX3JPIDgrR/8Pw26Skm19KCn5+TbKDfJwRaXpcjHQy5Ye7TuYZAAD//wnmPLTfAAAA" time=2026-03-23T23:56:03.840Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T23:56:03.841Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed pre-install: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4590 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4586 (merged) | 1 | provision (1)Unauthorized Unauthorized Unauthorized Unauthorized Show raw failuretime=2026-03-23T23:25:49.144Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-23T23:25:49.714Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:490c3b795cb12bd9e5a3680bb7dabd7ea02799c30b14ed960611517c5d808e6b imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[cpu:50m memory:100Mi] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-03-23T23:25:49.714Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-23T23:25:49.961Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy namespace=secret-sync-controller time=2026-03-23T23:25:50.287Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy dryRun=false time=2026-03-23T23:25:50.287Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-23T23:25:50.435Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-23T23:25:59.155Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy err="Unauthorized\nUnauthorized\nUnauthorized\nUnauthorized" time=2026-03-23T23:25:59.155Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-23T23:25:59.155Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-23T23:26:04.109Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy release=secret-sync-controller namespace=secret-sync-controller status=pending-install description="Initial install underway" values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:490c3b795cb12bd9e5a3680bb7dabd7ea02799c30b14ed960611517c5d808e6b imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[cpu:50m memory:100Mi] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-03-23T23:26:04.110Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/7SVT4vbPBDG7/4Ug08bsJaQsC+8KXvYlp4KpbSlhy09yNKDVxt5xkhKQiAfvjjOH7NN0hTik2A08/xG6JEm80gUEGURDCI9ktVJJ1163En5CpM+ObazmILjqqAu9FnXOISwBG8jsdFmHx79zIiI8g9+ERPCV/HIC8ojTEBScc1GaVs7boN5cSF3mwrrkgQVdjsnCt47to6rXp2KSQI6lBFOQbxHULVmXaHTKo9Fe8k38DMiHtoiKHiY5IQPjfWPd0zfa//Q3lmdHFdPtnYxOuEv4p1Z/x1oAnSCWjS2XZpD1c2lk8zByoLXt6VYeNy6813L5gVmrsTbzpu3ZSy7UijNLElvb3sYgNcl/EDa3a3+g/bVz+mUMdWJd3Vr0h8+HQa6o8mKEeIZpw1DPm+8gXl9Hw6M6tvyFOobwtIZPBkjC05X/+0XfuLs17tsvigRGAnxYzu5Yrah1QsCKLkaMem6oRJpBTDdtW224bt8Mp78p8ZTNZl+n0xnk4fZw/g5H9H9PV3K+f85H42yDb2KY5o7to+OGaE3b4XpOGb78/XtYM021ARpd4+NFtSI7ZKvEynIS1VQjRh11UpKsAhUrnuHt4jmdwAAAP//lzNCfhUIAAA=" time=2026-03-23T23:26:04.110Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-23T23:26:04.110Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Unauthorized\nUnauthorized\nUnauthorized\nUnauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4589 (closed) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>. Show raw failuretime=2026-03-23T23:34:29.701Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-23T23:34:30.234Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-23T23:34:31.142Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=1274b489e405478a98e50259476f20a0959f9f26bb68b51343591eb230be2a5e
time=2026-03-23T23:35:01.387Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9211264-mgmt-1/providers/Microsoft.Resources/deployments/1274b489e405478a98e50259476f20a0959f9f26bb68b51343591eb230be2a5e/operationStatuses/08584272980148477728\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is 37ef272e2ef651f1be383cfc74698550.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #4513 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4586 (merged) | 1 | provision (1)create: failed to create: Unauthorized Show raw failuretime=2026-03-23T22:51:44.579Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies description="Step deploy-policies\n Kind: Helm\n" time=2026-03-23T22:51:45.140Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies values=map[] time=2026-03-23T22:51:45.140Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-23T22:51:45.384Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies namespace=open-cluster-management-policies time=2026-03-23T22:51:45.704Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies dryRun=false time=2026-03-23T22:51:45.704Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-23T22:51:45.832Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-23T22:51:49.330Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies err="create: failed to create: Unauthorized" time=2026-03-23T22:51:49.330Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-23T22:51:49.330Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-23T22:51:49.566Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies err="failed to get status for release policy: release: not found" time=2026-03-23T22:51:49.566Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-23T22:51:49.566Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies err="error running Helm release deployment Step, failed to deploy helm release: create: failed to create: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4584 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4482 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4589 (closed) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>. Show raw failuretime=2026-03-23T22:35:15.970Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-23T22:35:16.418Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-23T22:35:17.444Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=ee5c573b597439779ce725913807e1ec592a2cdb091cfb47e69e84f48758905a
time=2026-03-23T22:35:47.639Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0574848-mgmt-1/providers/Microsoft.Resources/deployments/ee5c573b597439779ce725913807e1ec592a2cdb091cfb47e69e84f48758905a/operationStatuses/08584273015687315026\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is ffb777f2900b52a28855ed2d9ce453f6.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4585 (merged) | 1 | provision (1)failed post-install: warning: Hook post-install multicluster-engine-local-cluster/templates/finalize.job.yaml failed: Unauthorized Show raw failuretime=2026-03-23T22:08:31.203Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-23T22:08:31.798Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5d1fc3770795996bc22512c28adf4623ecb8a69eec3018a88e34a87bcba3366a global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-23T22:08:31.798Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T22:08:31.871Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-23T22:08:32.020Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-23T22:08:32.020Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T22:08:32.160Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T22:08:59.219Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: warning: Hook post-install multicluster-engine-local-cluster/templates/finalize.job.yaml failed: Unauthorized" time=2026-03-23T22:08:59.219Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T22:08:59.219Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T22:09:03.131Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Release \"mce-config\" failed: failed post-install: warning: Hook post-install multicluster-engine-local-cluster/templates/finalize.job.yaml failed: Unauthorized" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5d1fc3770795996bc22512c28adf4623ecb8a69eec3018a88e34a87bcba3366a global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-23T22:09:03.132Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yUT2/aTBDG7/4UI59AwhEyel+1VDnQtKcordT0lKqH9e5gNqxnrNkFhMSHr/wPUxICpb0+nvnpeXbGEzkMIOh5JRo93IJRQQWVORxw9ow63FsyUx/EUj6CRvqiCtxLuEaqFV8q3cnDHxEAQHzvVj6gOAwzY5jumOY2j0cQO9bKJbr5/IrQtD8oUjmau0Z9xFBVLrYlil/YeUiK+nuBFLpOX1V07Y8oa6txpjWvqO5d7v0kqjKUaKYg7BxKsk6rimLlgm1hCVJuCU/jctFJyc7qbYvz6gyjjfKNHX60ZCzVr8ElUpfgINT0vN3ruLnoaSsLO8z6jo53BKqCvgh71PpG6hO0ywEzY77SJywdb6sA/R41ThbsA5rqdVr5/AQe9q9Rs2tbvEYhRRq7nHNRBW5YlodP0+90byN3nCl3YiD/fMJ/MNorCU3+1tHrtA5zcjUu2InjZTjT0i/AX//Lv6Ne+O9R13JK4VLlKvBJQvTzQ7RcZSiEAf3n6oz6aAebBQpCsAX6oIoSMgwbRIKBUQEreRCn4/T/ZDxJ0sn3NJ2O300n6VM8hJsbeKPmv/dP8XAY7eCZLcHSkrm1RCgHx58J+pt/eOyPr3y0g1K4+tobHUHJpim+DDICx/kICvRe5RWSxaBAtj0Ib9DrXwEAAP//r5hklaIGAAA=" time=2026-03-23T22:09:03.133Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T22:09:03.134Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: warning: Hook post-install multicluster-engine-local-cluster/templates/finalize.job.yaml failed: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4556 (open) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T21:43:19.090Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T21:43:37.165Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=a4d3a2a1-c8bb-4792-8cae-355b5030e10c && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=bf8ce325-899b-4289-8318-606037f6dfde && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj2975104.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj2975104.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j2975104 -n arohcp-prow-dbcs-j2975104 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2975104/providers/Microsoft.Network/dnszones/j2975104.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j2975104-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j2975104-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j2975104-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"51a31a28-8560-4ccd-b603-6213d8bf44ac\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j2975104-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2975104-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j2975104-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: create: failed to create: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4586 (merged) | 1 | provision (1)failed to run ARM step: failed to poll previously-existing deployment: timed out waiting for pre-existing deployment to finish Show raw failuretime=2026-03-23T21:42:27.275Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=management step=output description="Step output\n Kind: ARM\n Template: ../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-03-23T21:42:27.852Z level=DEBUG+2 msg="Waiting for existing deployment to complete" serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=management step=output deployment=14278ec7016f40fafde1317e05d76b8961b538d2c814ee2123a7c263f16c5d3a time=2026-03-23T21:45:29.019Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=management step=output err="failed to run ARM step: failed to poll previously-existing deployment: timed out waiting for pre-existing deployment to finish" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4589 (closed) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>. Show raw failuretime=2026-03-23T21:20:54.796Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-23T21:20:55.446Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-23T21:20:56.735Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=18cad54200e7607a0caa73878ae72d675e10111a6f936a591fd9daeaa39784c0
time=2026-03-23T21:21:26.980Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5672064-mgmt-1/providers/Microsoft.Resources/deployments/18cad54200e7607a0caa73878ae72d675e10111a6f936a591fd9daeaa39784c0/operationStatuses/08584273060294549784\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is 7f32bc67708b533c99857e6f580b6d47.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4554 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4586 (merged) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T21:07:24.361Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T21:08:34.422Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=55915c73-d067-49b5-a2bf-3413afbb0134 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=4ffbf161-8d3e-4a13-9066-782cc4d019d9 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj6813056.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj6813056.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j6813056 -n arohcp-prow-dbcs-j6813056 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6813056/providers/Microsoft.Network/dnszones/j6813056.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j6813056-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j6813056-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j6813056-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"6823beeb-33ae-4516-aab1-7798c8f63116\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j6813056-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6813056-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j6813056-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #4556 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4482 (merged) | 1 | provision (1)failed post-install: an error on the server ("Internal Server Error: \\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\": Post \\"<url>": proxy error from localhost:9443 while dialing Show raw failuretime=2026-03-23T21:04:32.625Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-23T21:04:33.148Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5d1fc3770795996bc22512c28adf4623ecb8a69eec3018a88e34a87bcba3366a global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-23T21:04:33.148Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T21:04:33.214Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-23T21:04:33.345Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-23T21:04:33.345Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T21:04:33.472Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T21:05:57.245Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69c1a715a03f4d0001c61afb.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69c1a715a03f4d0001c61afb.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" time=2026-03-23T21:05:57.245Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T21:05:57.245Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T21:05:57.394Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine/secrets?labelSelector=name%3Dmce-config%2Cowner%3Dhelm\\\": Post \\\"https://guard.69c1a715a03f4d0001c61afb.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69c1a715a03f4d0001c61afb.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get secrets)" time=2026-03-23T21:05:57.394Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-23T21:05:57.394Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69c1a715a03f4d0001c61afb.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69c1a715a03f4d0001c61afb.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4588 (merged) | 1 | provision (1)cluster reachability check failed: kubernetes cluster unreachable: the server has asked for the client to provide credentials Show raw failuretime=2026-03-23T20:53:47.684Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-23T20:53:48.939Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b7c83e5834c5e97ef17af96ddbcba9ed7062371a1db1a46171ff7ae1ae137aec] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b7bd3a6bf3160b82af89c345c58b1fd3a936801c8c65fef7ed2111dc71c766a3]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:7ec6de8f-1c00-4a8f-9c52-258e1fa09011]] prometheusSpec:map[externalLabels:map[cluster:prow-j5920768-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j5920768-mgmt-1-bvfq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-80a0578c8edd4126b2767f29342b397e/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:eb5dcb5c4f6e229d6dfe17b492291e06bea7ba19f080d48a7afc663e4420386c] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j5920768-mgmt-1-bvfq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-30c0409761ed4d3ca84fd154689b07a0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-23T20:53:48.939Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T20:53:49.266Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-23T20:53:49.671Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-23T20:53:49.671Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T20:53:49.874Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T20:53:50.460Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="cluster reachability check failed: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-23T20:53:50.460Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T20:53:50.460Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T20:53:50.679Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed to get status for release arohcp-monitor: release: not found" time=2026-03-23T20:53:50.679Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T20:53:50.679Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: cluster reachability check failed: kubernetes cluster unreachable: the server has asked for the client to provide credentials" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4587 (merged) | 1 | provision (1)failed to apply namespace prometheus: Unauthorized Show raw failuretime=2026-03-23T19:52:45.211Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-23T19:52:45.790Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b7c83e5834c5e97ef17af96ddbcba9ed7062371a1db1a46171ff7ae1ae137aec] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b7bd3a6bf3160b82af89c345c58b1fd3a936801c8c65fef7ed2111dc71c766a3]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:e3ec7c4a-d67f-417f-a298-01edac1d5653]] prometheusSpec:map[externalLabels:map[cluster:prow-j0167808-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j0167808-mgmt-1-ppdq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-a90f9d625b434c29813a131df9e03268/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:eb5dcb5c4f6e229d6dfe17b492291e06bea7ba19f080d48a7afc663e4420386c] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j0167808-mgmt-1-ppdq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-368fc8df388b4b90a814030d03dd4f3e/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-23T19:52:45.790Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T19:52:46.286Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace prometheus: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4586 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4393 (merged) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T19:20:44.752Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T19:22:36.961Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=9fc8d3f9-d85d-4b12-b7f3-548e1bd147a2 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=f610aae7-53b6-41d5-a4d0-dc6cea834431 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj8012288.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj8012288.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j8012288 -n arohcp-prow-dbcs-j8012288 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8012288/providers/Microsoft.Network/dnszones/j8012288.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j8012288-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j8012288-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j8012288-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"542ca1fd-842a-439e-8779-f6f37f986389\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j8012288-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8012288-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j8012288-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4584 (merged) | 1 | provision (1)failed to apply namespace openshift-route-monitor-operator: Unauthorized Show raw failuretime=2026-03-23T18:50:04.100Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-23T18:50:04.664Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-23T18:50:04.664Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-23T18:50:05.147Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace openshift-route-monitor-operator: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4482 (merged) | 1 | e2e (1)ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:125]: Unexpected error:
<*fmt.wrapError | 0xc0000dcb00>:
failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000912ff0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"b163422a-6a86-40c3-af8a-af46b3a6eb36",
],
"Date": [
"Mon, 23 Mar 2026 19:13:37 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["64"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0000dc580>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7310080.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:125]: Unexpected error:
<*fmt.wrapError | 0xc0000dcb00>:
failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000912ff0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"b163422a-6a86-40c3-af8a-af46b3a6eb36",
],
"Date": [
"Mon, 23 Mar 2026 19:13:37 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["64"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0000dc580>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7310080.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4585 (merged) | 4 | e2e (4)ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:167]: Unexpected error:
<*fmt.wrapError | 0xc000e04780>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-rl8trq" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000e04760>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00155d230>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"df7e757d-5e64-40a7-ade1-a33a21eb48eb",
],
"Date": [
"Mon, 23 Mar 2026 18:53:15 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2823"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc000e04600>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:167]: Unexpected error:
<*fmt.wrapError | 0xc000e04780>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-rl8trq" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000e04760>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00155d230>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"df7e757d-5e64-40a7-ade1-a33a21eb48eb",
],
"Date": [
"Mon, 23 Mar 2026 18:53:15 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2823"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc000e04600>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
<*fmt.wrapError | 0xc0009d0dc0>:
failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-pv6mwr" to finish updating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-pv6mwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-pv6mwr\" to finish updating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-pv6mwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00095d320>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2860"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"cf637623-342c-4b03-8376-ec174e18500b",
],
"Date": [
"Mon, 23 Mar 2026 18:53:15 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0009d0d00>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-pv6mwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T18:52:47Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (g...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
<*fmt.wrapError | 0xc0009d0dc0>:
failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-pv6mwr" to finish updating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-pv6mwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-pv6mwr\" to finish updating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-pv6mwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00095d320>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2860"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"cf637623-342c-4b03-8376-ec174e18500b",
],
"Date": [
"Mon, 23 Mar 2026 18:53:15 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0009d0d00>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-pv6mwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T18:52:47Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (g...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredAzureCLICredential: signal: killed Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:152]: Unexpected error:
<*azidentity.AuthenticationFailedError | 0xc000995b90>:
AzureCLICredential: signal: killed
{
RawResponse: nil,
credType: "AzureCLICredential",
message: "signal: killed",
omitResponse: false,
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:152]: Unexpected error:
<*azidentity.AuthenticationFailedError | 0xc000995b90>:
AzureCLICredential: signal: killed
{
RawResponse: nil,
credType: "AzureCLICredential",
message: "signal: killed",
omitResponse: false,
}
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:139]: Unexpected error:
<*fmt.wrapError | 0xc0008c0c00>:
failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-rhqc66" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"ephemeral-np\" for cluster \"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rhqc66\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000937440>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"e2e35a32-25dd-4716-bde4-6f5b2d4421c3",
],
"Date": [
"Mon, 23 Mar 2026 18:53:09 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["3006"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0008c0b40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T18:52:45Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:139]: Unexpected error:
<*fmt.wrapError | 0xc0008c0c00>:
failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-rhqc66" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"ephemeral-np\" for cluster \"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rhqc66\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000937440>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"e2e35a32-25dd-4716-bde4-6f5b2d4421c3",
],
"Date": [
"Mon, 23 Mar 2026 18:53:09 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["3006"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0008c0b40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T18:52:45Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4586 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4505 (merged) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T18:04:49.476Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T18:05:15.919Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy step=deploy error="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=32fcf37b-af9e-40c7-9ca0-7b55a80a7367 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=833a9afd-6f6e-43e3-8229-44e0623dfc7f && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj0974466.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj0974466.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j0974466 -n arohcp-prow-dbcs-j0974466 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0974466/providers/Microsoft.Network/dnszones/j0974466.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j0974466-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j0974466-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j0974466-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"afa01b7c-2266-4069-b0ee-3f2e2e7482d0\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j0974466-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0974466-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j0974466-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: 1 error occurred:\n\t* Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.40:15017, code 500: 500 Internal Server Error\n\n\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
time=2026-03-23T18:06:15.920Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T18:06:30.844Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=32fcf37b-af9e-40c7-9ca0-7b55a80a7367 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=833a9afd-6f6e-43e3-8229-44e0623dfc7f && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj0974466.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj0974466.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j0974466 -n arohcp-prow-dbcs-j0974466 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0974466/providers/Microsoft.Network/dnszones/j0974466.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j0974466-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j0974466-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j0974466-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"afa01b7c-2266-4069-b0ee-3f2e2e7482d0\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j0974466-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0974466-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j0974466-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service unchanged\nnamespace/clusters-service not labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nError: UPGRADE FAILED: could not get information about the resource: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4473 (merged) | 1 | provision (1)"mce-crds" has no deployed releases Show raw failuretime=2026-03-23T18:21:44.825Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds description="Step deploy-mce-crds\n Kind: Helm\n" time=2026-03-23T18:21:45.294Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds values=map[] time=2026-03-23T18:21:45.294Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-23T18:21:45.837Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds namespace=multicluster-engine time=2026-03-23T18:21:46.324Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds dryRun=false time=2026-03-23T18:21:46.324Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-23T18:21:46.884Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds numVersions=0 time=2026-03-23T18:21:47.081Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="\"mce-crds\" has no deployed releases" time=2026-03-23T18:21:47.081Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-23T18:21:47.081Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-23T18:21:47.212Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="failed to get status for release mce-crds: release: not found" time=2026-03-23T18:21:47.212Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-23T18:21:47.212Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="error running Helm release deployment Step, failed to deploy helm release: \"mce-crds\" has no deployed releases" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4585 (merged) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T17:20:09.109Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T17:20:27.463Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=0f8a3c87-daee-4534-8adb-978d10f3fbc3 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=a9d7c021-5881-4d2e-9250-9944da142893 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3637376.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3637376.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3637376 -n arohcp-prow-dbcs-j3637376 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3637376/providers/Microsoft.Network/dnszones/j3637376.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3637376-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3637376-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j3637376-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"25a5d469-6151-4fcb-b6ab-3254f7c5f6f8\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3637376-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3637376-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3637376-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: could not get server version from Kubernetes: the server has asked for the client to provide credentials\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4584 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4518 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4556 (open) | 4 | e2e (4)ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
<*fmt.wrapError | 0xc00011ef20>:
failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rsp6lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g75n2h/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rsp6lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g75n2h/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001440510>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"fad802c6-cf1f-464a-8144-b9fa410d06da",
],
"Date": [
"Mon, 23 Mar 2026 17:14:23 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["76"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc00011eca0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rsp6lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g75n2h/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:14:13...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
<*fmt.wrapError | 0xc00011ef20>:
failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rsp6lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g75n2h/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rsp6lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g75n2h/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001440510>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"fad802c6-cf1f-464a-8144-b9fa410d06da",
],
"Date": [
"Mon, 23 Mar 2026 17:14:23 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["76"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc00011eca0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rsp6lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g75n2h/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:14:13...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:115]: Unexpected error:
<*fmt.wrapError | 0xc0006be500>:
failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-8ftcvf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-8ftcvf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001514810>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["107"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"6b966dbe-edf7-4bda-873e-881fcfddda7a",
],
"Date": [
"Mon, 23 Mar 2026 17:14:18 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0006be440>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-8ftcvf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Authorization": ["redacted"],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:14:09Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:115]: Unexpected error:
<*fmt.wrapError | 0xc0006be500>:
failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-8ftcvf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-8ftcvf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001514810>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["107"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"6b966dbe-edf7-4bda-873e-881fcfddda7a",
],
"Date": [
"Mon, 23 Mar 2026 17:14:18 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0006be440>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-8ftcvf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Authorization": ["redacted"],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:14:09Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc000984360>:
failed to create HCP cluster cluster-zstream-4-21-9pnhv4: failed waiting for cluster="cluster-zstream-4-21-9pnhv4" in resourcegroup="rg-zstream-upgrade-4-21-7nx4d8" to finish creating: GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-7nx4d8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9pnhv4
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-21-9pnhv4: failed waiting for cluster=\"cluster-zstream-4-21-9pnhv4\" in resourcegroup=\"rg-zstream-upgrade-4-21-7nx4d8\" to finish creating: GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-7nx4d8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9pnhv4\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000984300>{
msg: "failed waiting for cluster=\"cluster-zstream-4-21-9pnhv4\" in resourcegroup=\"rg-zstream-upgrade-4-21-7nx4d8\" to finish creating: GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-7nx4d8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9pnhv4\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0013002d0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["3511"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"5ffba58e-5961-48d0-9dab-a5a07d23df06",
],
"Date": [
"Mon, 23 Mar 2026 17:23:13 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000984020>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstr...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc000984360>:
failed to create HCP cluster cluster-zstream-4-21-9pnhv4: failed waiting for cluster="cluster-zstream-4-21-9pnhv4" in resourcegroup="rg-zstream-upgrade-4-21-7nx4d8" to finish creating: GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-7nx4d8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9pnhv4
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-21-9pnhv4: failed waiting for cluster=\"cluster-zstream-4-21-9pnhv4\" in resourcegroup=\"rg-zstream-upgrade-4-21-7nx4d8\" to finish creating: GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-7nx4d8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9pnhv4\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000984300>{
msg: "failed waiting for cluster=\"cluster-zstream-4-21-9pnhv4\" in resourcegroup=\"rg-zstream-upgrade-4-21-7nx4d8\" to finish creating: GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-7nx4d8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9pnhv4\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0013002d0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["3511"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"5ffba58e-5961-48d0-9dab-a5a07d23df06",
],
"Date": [
"Mon, 23 Mar 2026 17:23:13 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000984020>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstr...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:418]: Unexpected error:
<*exported.ResponseError | 0xc0010b8000>:
GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-snv4ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["2610"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"b7d20e26-db8e-4737-b04f-c04d91aaed58",
],
"Date": [
"Mon, 23 Mar 2026 17:09:57 GMT",
],
"Content-Length": ["107"],
},
Body: <*exported.nopClosingBytesReader | 0xc0009c4f80>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-snv4ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:09:36Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
DidResume: false,
CipherSuite: 4865,
CurveID: 29,
NegotiatedProtocol: "h2",
NegotiatedProtocolIsMutual: true,
ServerName: "rp.j2996352.hcpsvc.osadev.cloud",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:418]: Unexpected error:
<*exported.ResponseError | 0xc0010b8000>:
GET https://rp.j2996352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-snv4ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["2610"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"b7d20e26-db8e-4737-b04f-c04d91aaed58",
],
"Date": [
"Mon, 23 Mar 2026 17:09:57 GMT",
],
"Content-Length": ["107"],
},
Body: <*exported.nopClosingBytesReader | 0xc0009c4f80>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-snv4ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:09:36Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "rp.j2996352.hcpsvc.osadev.cloud",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
DidResume: false,
CipherSuite: 4865,
CurveID: 29,
NegotiatedProtocol: "h2",
NegotiatedProtocolIsMutual: true,
ServerName: "rp.j2996352.hcpsvc.osadev.cloud",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4585 (merged) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T16:49:01.435Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T16:49:18.472Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=a1860c91-e1c4-4628-8c7a-e39af585e22f && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=68990b8b-945a-40a4-ad14-c0f8ee107135 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj9011712.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj9011712.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j9011712 -n arohcp-prow-dbcs-j9011712 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9011712/providers/Microsoft.Network/dnszones/j9011712.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j9011712-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j9011712-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j9011712-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"40329ede-b468-40bf-8e42-41153ed76e00\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j9011712-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9011712-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j9011712-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unable to continue with install: could not get information about the resource Deployment \"postgres-breakglass\" in namespace \"clusters-service\": Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4579 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4585 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4457 (open) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T16:33:04.940Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T16:34:25.561Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=8ae5b76d-6bd0-44ed-9ecc-d9f925927be3 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=8009bd65-fd90-4d10-9d07-aca4690540d4 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj8805376.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj8805376.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j8805376 -n arohcp-prow-dbcs-j8805376 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8805376/providers/Microsoft.Network/dnszones/j8805376.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j8805376-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j8805376-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j8805376-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"794c7e96-3278-4db5-9dce-f6a403211e98\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j8805376-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8805376-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j8805376-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4393 (merged) | 1 | provision (1)failed to apply namespace multicluster-engine: Unauthorized Show raw failuretime=2026-03-23T16:39:59.671Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds description="Step deploy-mce-crds\n Kind: Helm\n" time=2026-03-23T16:40:00.185Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds values=map[] time=2026-03-23T16:40:00.185Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-23T16:40:00.822Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace multicluster-engine: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4584 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4563 (open) | 2 | e2e (2)ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error:
<*fmt.wrapError | 0xc00099eae0>:
failed to start credential request: POST https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-chpshm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-chpshm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00092b800>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"244bd667-fd47-429e-a668-8676c9666731",
],
"Date": [
"Mon, 23 Mar 2026 17:05:44 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["51"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc00099ea20>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7146752.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-chpshm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:05:30Z\"}",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error:
<*fmt.wrapError | 0xc00099eae0>:
failed to start credential request: POST https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-chpshm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-chpshm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00092b800>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"244bd667-fd47-429e-a668-8676c9666731",
],
"Date": [
"Mon, 23 Mar 2026 17:05:44 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["51"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc00099ea20>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7146752.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-chpshm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:05:30Z\"}",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
<*fmt.wrapError | 0xc0008e8c00>:
failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-j84dxz" to finish updating: GET https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-j84dxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-j84dxz\" to finish updating: GET https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-j84dxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008acf90>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"d119aea6-57cb-4bf2-9e82-0f2727d1ef38",
],
"Date": [
"Mon, 23 Mar 2026 17:11:05 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2834"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc0008e8b00>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7146752.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-j84dxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:10:44Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (g...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
<*fmt.wrapError | 0xc0008e8c00>:
failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-j84dxz" to finish updating: GET https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-j84dxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-j84dxz\" to finish updating: GET https://rp.j7146752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-j84dxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008acf90>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"d119aea6-57cb-4bf2-9e82-0f2727d1ef38",
],
"Date": [
"Mon, 23 Mar 2026 17:11:05 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2834"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc0008e8b00>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7146752.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-j84dxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-one-node",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T17:10:44Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (g...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4583 (merged) | 1 | provision (1)failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name="fluentbit"].limits: field not declared in schema Show raw failuretime=2026-03-23T16:16:36.862Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit description="Step arobit\n Kind: Helm\n" time=2026-03-23T16:16:37.551Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:prow-j8017664-mgmt-1 clusterType:mgmt dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:33bed43b09e3d2384c7661c74f6ed79ca53b0c25a3e28d77f139a7b9d50fbf1a pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit]] geoShortId:us kusto:map[enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net serviceLogsDatabase:ServiceLogs] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan: clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-prow-j8017664-mgmt-1 rpAccountName:placeholder rpNamespace: rpSan: rpSecretName:rplogs] image:map[digest:sha256:18d5476700adcd40e9ee2bc1cb62a1150ac1b5e696d7bad6adeea4da88c615db pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:27a07e3a-f481-439c-8e7a-e2a4ed160dda region:westus3 secretProvider:map[keyVault:ah-prow-mg-j8017664-1] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]" time=2026-03-23T16:16:37.551Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-03-23T16:16:37.865Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit namespace=arobit time=2026-03-23T16:16:38.161Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit dryRun=false time=2026-03-23T16:16:38.161Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-03-23T16:16:38.293Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-03-23T16:16:43.187Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit err="failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name=\"fluentbit\"].limits: field not declared in schema" time=2026-03-23T16:16:43.187Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-03-23T16:16:43.187Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-03-23T16:16:43.792Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit release=arobit namespace=arobit status=failed description="Release \"arobit\" failed: failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name=\"fluentbit\"].limits: field not declared in schema" values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:prow-j8017664-mgmt-1 clusterType:mgmt dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:33bed43b09e3d2384c7661c74f6ed79ca53b0c25a3e28d77f139a7b9d50fbf1a pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit]] geoShortId:us kusto:map[enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net serviceLogsDatabase:ServiceLogs] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan: clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-prow-j8017664-mgmt-1 rpAccountName:placeholder rpNamespace: rpSan: rpSecretName:rplogs] image:map[digest:sha256:18d5476700adcd40e9ee2bc1cb62a1150ac1b5e696d7bad6adeea4da88c615db pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:27a07e3a-f481-439c-8e7a-e2a4ed160dda region:westus3 secretProvider:map[keyVault:ah-prow-mg-j8017664-1] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]" time=2026-03-23T16:16:43.793Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5SRwU7CQBCG732KSU80KQSoIQbDQY0ngwf1hPGw3R3qQjvTzE4hJjy8qaCgEdHrt98/mZ0/KlFBMHAjFgNMwBk1avISO5wv0OqtJzcOKp6KFLbozlT4iXCF9E5CbewHTp4iAID4mmnui6mp4xRiI5x77c5Z1kYcyp7F6U4vm6Ao91ziscAP6pUn56k4lXhAWXl7dPDXTXbylMkry78yl9ZyQ3oiEz1fRMsmRyFUDDftEUO0gfULCoL6CoOaqoYcdY1I0HFGscWdeNgfjrr9rDvMHgej8WA0zs5ncQK9HvzinGWzOEmiDSzYEyw9uYknQjmongn2jR9W/b3jaAO1cPu6XzSFmt1W/tuQFEouUqgwBFO0I7k9EOSvB593GOxbAAAA//8h+DJdoAIAAA==" time=2026-03-23T16:16:43.793Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-03-23T16:16:43.794Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name=\"fluentbit\"].limits: field not declared in schema" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4473 (merged) | 1 | provision (1)tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\ Show raw failuretime=2026-03-23T16:20:04.257Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-23T16:20:05.290Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b7c83e5834c5e97ef17af96ddbcba9ed7062371a1db1a46171ff7ae1ae137aec] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b7bd3a6bf3160b82af89c345c58b1fd3a936801c8c65fef7ed2111dc71c766a3]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:40959f79-762b-4855-8ace-e32c9a85699e]] prometheusSpec:map[externalLabels:map[cluster:prow-j5098368-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:eb5dcb5c4f6e229d6dfe17b492291e06bea7ba19f080d48a7afc663e4420386c] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j5098368-svc-ucq1.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-d742c6875b444f0288b9c15c476b06c1/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-23T16:20:05.290Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-23T16:20:05.382Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace prometheus: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/prometheus?fieldManager=templatize&force=false\\\": Post \\\"https://guard.69c164e47a8970000186b27d.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (patch namespaces prometheus)" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4554 (merged) | 1 | provision (1)failed post-install: warning: Hook post-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized Show raw failuretime=2026-03-23T16:18:27.340Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-23T16:18:28.018Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b7c83e5834c5e97ef17af96ddbcba9ed7062371a1db1a46171ff7ae1ae137aec] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b7bd3a6bf3160b82af89c345c58b1fd3a936801c8c65fef7ed2111dc71c766a3]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:eb6fef22-193d-40a1-b6b4-e1f8efbd4b8a]] prometheusSpec:map[externalLabels:map[cluster:prow-j2292736-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j2292736-mgmt-1-2jsb.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-c53a8f96f02444a8877e72857b477fc0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:eb5dcb5c4f6e229d6dfe17b492291e06bea7ba19f080d48a7afc663e4420386c] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j2292736-mgmt-1-2jsb.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8b244679ca374250b457b2ec0755c578/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-23T16:18:28.019Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T16:18:28.313Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-23T16:18:28.658Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-23T16:18:28.658Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T16:18:28.831Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T16:20:41.651Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: warning: Hook post-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized" time=2026-03-23T16:20:41.651Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T16:20:41.651Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T16:20:42.204Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed to get status for release arohcp-monitor: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-23T16:20:42.204Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-23T16:20:42.204Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: warning: Hook post-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4556 (open) | 1 | provision (1)make[2]: *** [Makefile:32: deploy] Error 1 Show raw failuretime=2026-03-23T16:02:48.155Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T16:03:26.993Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=84b71e41-ec7c-4545-89ac-49bea8759c70 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=23f0e43a-8fea-4e65-b661-1bc7c496f80e && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj7746560.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj7746560.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j7746560 -n arohcp-prow-dbcs-j7746560 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7746560/providers/Microsoft.Network/dnszones/j7746560.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j7746560-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j7746560-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j7746560-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"9b43dd6b-be77-41f2-83c5-72b7b97417ee\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j7746560-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7746560-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j7746560-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4507 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4505 (merged) | 1 | provision (1)failed to apply namespace monitoring: Unauthorized Show raw failuretime=2026-03-23T15:33:21.717Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events description="Step kube-events\n Kind: Helm\n" time=2026-03-23T15:33:22.530Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events values="map[enabled:true image:map[digest:sha256:f907f0eb97bdf12502907c5003845276d41662736d7a5462ae597eb74e337bc9 registry:kubernetesshared.azurecr.io repository:shared/kube-events]]" time=2026-03-23T15:33:22.530Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events time=2026-03-23T15:33:23.353Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace monitoring: Unauthorized" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4510 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4576 (merged) | 1 | e2e (1)ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
<*fmt.wrapError | 0xc000f54b60>:
failed to start credential request: POST https://rp.j3617920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-lqctzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j3617920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-lqctzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001561b30>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["307"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"437b836e-e757-4253-8dff-267c064c23e3",
],
"Date": [
"Mon, 23 Mar 2026 15:19:03 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000f54aa0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3617920.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-lqctzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T15:18:54Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
<*fmt.wrapError | 0xc000f54b60>:
failed to start credential request: POST https://rp.j3617920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-lqctzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j3617920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-lqctzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001561b30>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["307"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"437b836e-e757-4253-8dff-267c064c23e3",
],
"Date": [
"Mon, 23 Mar 2026 15:19:03 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000f54aa0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3617920.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-lqctzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T15:18:54Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4581 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4580 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4257 (merged) | 1 | e2e (1)UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc000f91320>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-9trnk9" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-9trnk9, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-9trnk9\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-9trnk9, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e729f0>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-9trnk9",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc000f91320>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-9trnk9" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-9trnk9, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-9trnk9\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-9trnk9, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e729f0>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-9trnk9",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4004 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4579 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4494 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4576 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4566 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4553 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4494 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4510 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4575 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4569 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4566 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4575 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4575 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4574 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4574 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code BadRequest; detail message Preflight validation check for resource(s) for container service prow-j8926976-svc in resource group <resource-group> failed. Message: Requested change in revisions is not allowed. Reason: Change from current revision(s) [] to requested ["asm-1-26" "asm-1-28 Show raw failuretime=2026-03-23T03:31:28.344Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-23T03:31:30.065Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-23T03:31:31.832Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=06e0035c2cc05044b786bcb1bfbff81839e23ae5acbb189a590b7575bbc787f8
time=2026-03-23T03:36:03.164Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8926976-svc/providers/Microsoft.Resources/deployments/06e0035c2cc05044b786bcb1bfbff81839e23ae5acbb189a590b7575bbc787f8/operationStatuses/08584273701947108281\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\r\\n \\\"message\\\": \\\"The template deployment 'cluster-5ynmvygxl445g' is not valid according to the validation procedure. The tracking id is '51d60b8b-f5d8-4d36-b610-bfb1ddf3c550'. See inner errors for details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"BadRequest\\\",\\r\\n \\\"message\\\": \\\"Preflight validation check for resource(s) for container service prow-j8926976-svc in resource group hcp-underlay-prow-j8926976-svc failed. Message: Requested change in revisions is not allowed. Reason: Change from current revision(s) [] to requested [\\\\\\\"asm-1-26\\\\\\\" \\\\\\\"asm-1-28\\\\\\\"] is not valid. Please verify current revision is not empty and requested revision is returned in 'az aks mesh get-upgrades' output.. Details: \\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4574 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code BadRequest; detail message Preflight validation check for resource(s) for container service prow-j0034176-svc in resource group <resource-group> failed. Message: Requested change in revisions is not allowed. Reason: Change from current revision(s) [] to requested ["asm-1-26" "asm-1-28 Show raw failuretime=2026-03-23T02:50:25.703Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-23T02:50:27.607Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-23T02:50:29.102Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4dcbe397d7cb9641f716e90b063fddfcc8005b59dc856eb846fa9e757aa8eeab
time=2026-03-23T02:54:30.280Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0034176-svc/providers/Microsoft.Resources/deployments/4dcbe397d7cb9641f716e90b063fddfcc8005b59dc856eb846fa9e757aa8eeab/operationStatuses/08584273726571660425\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\r\\n \\\"message\\\": \\\"The template deployment 'cluster-egnrtjj2oqpg6' is not valid according to the validation procedure. The tracking id is '5b0f0640-ef01-494d-99f4-125dc24c7413'. See inner errors for details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"BadRequest\\\",\\r\\n \\\"message\\\": \\\"Preflight validation check for resource(s) for container service prow-j0034176-svc in resource group hcp-underlay-prow-j0034176-svc failed. Message: Requested change in revisions is not allowed. Reason: Change from current revision(s) [] to requested [\\\\\\\"asm-1-26\\\\\\\" \\\\\\\"asm-1-28\\\\\\\"] is not valid. Please verify current revision is not empty and requested revision is returned in 'az aks mesh get-upgrades' output.. Details: \\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4463 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4570 (merged) | 0 | n/a |