No runs match your search.
Environment: DEV
| Time (UTC) | Runtime | Job | Region | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|---|---|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4545 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4548 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4473 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4536 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4535 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4547 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4457 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4513 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4305 (closed) | 1 | e2e (1)expected mirror source policy "AllowContactingSource" for source "fake-source.example.com/fake" (found: ) Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:159]: Timed out after 60.001s.
ImageDigestMirrorSet CRDs should exist on the hosted cluster
Expected success, but got an error:
<*errors.errorString | 0xc000e36110>:
IDMS verification failed:
expected mirror source policy "AllowContactingSource" for source "fake-source.example.com/fake" (found: )
{
s: "IDMS verification failed:\nexpected mirror source policy \"AllowContactingSource\" for source \"fake-source.example.com/fake\" (found: )",
}
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:159]: Timed out after 60.001s.
ImageDigestMirrorSet CRDs should exist on the hosted cluster
Expected success, but got an error:
<*errors.errorString | 0xc000e36110>:
IDMS verification failed:
expected mirror source policy "AllowContactingSource" for source "fake-source.example.com/fake" (found: )
{
s: "IDMS verification failed:\nexpected mirror source policy \"AllowContactingSource\" for source \"fake-source.example.com/fake\" (found: )",
} |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4484 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4425 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4425 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4234 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #3973 (merged) | 1 | e2e (1)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
<*fmt.wrapErrors | 0xc000faad50>:
failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-fkdqkt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded
{
msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<*fmt.wrapErrors | 0xc000faad20>{
msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
<*fmt.wrapErrors | 0xc000faad50>:
failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-fkdqkt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded
{
msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<*fmt.wrapErrors | 0xc000faad20>{
msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4537 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4508 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4425 (merged) | 1 | provision (1)resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress context deadline exceeded Show raw failuretime=2026-03-19T20:03:31.260Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-19T20:03:31.868Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy values="map[adminApi:map[namespace:aro-hcp-admin-api serviceAccount:admin-api] adminApiHost:admin.j7131520.hcpsvc.osadev.cloud apiBindPort:8443 audit:map[connectSocket:false] azure:map[clientId:9ae2dce2-36fd-4de1-9388-9b4bdd3466c8 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[databaseName:arohcpprow-rp-j7131520 databaseUrl:https://arohcpprow-rp-j7131520.documents.azure.com:443/ frontendMiClientId:3e51afb0-72c4-4214-8b19-e0e34a17f6b1 location:westus3] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:frontend-cert-prow-j7131520] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-l6h99n9b/pipeline@sha256:b84a27900c8a20ec255c2885caeb02d0d956d9ae792d8dcfba0223f93828f499 replicas:2 zoneCount:4] exitOnPanic:true mise:map[deploy:false policyLabel:ARM Policy] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-l6h99n9b/pipeline:pull workloadIdentityClientId:20c85050-271a-4f6d-b919-9e226971e33c workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:frontend workloadIdentityClientId:3e51afb0-72c4-4214-8b19-e0e34a17f6b1 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp] virtualService:map[host:rp.j7131520.hcpsvc.osadev.cloud]]" time=2026-03-19T20:03:31.869Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy time=2026-03-19T20:03:32.044Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy namespace=aro-hcp time=2026-03-19T20:03:32.363Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy dryRun=false time=2026-03-19T20:03:32.363Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy time=2026-03-19T20:03:32.472Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy time=2026-03-19T20:08:38.325Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-19T20:08:38.325Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy time=2026-03-19T20:08:38.325Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy time=2026-03-19T20:08:41.331Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy release=aro-hcp-frontend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-frontend-dev\" failed: resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[adminApi:map[namespace:aro-hcp-admin-api serviceAccount:admin-api] adminApiHost:admin.j7131520.hcpsvc.osadev.cloud apiBindPort:8443 audit:map[connectSocket:false] azure:map[clientId:9ae2dce2-36fd-4de1-9388-9b4bdd3466c8 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[databaseName:arohcpprow-rp-j7131520 databaseUrl:https://arohcpprow-rp-j7131520.documents.azure.com:443/ frontendMiClientId:3e51afb0-72c4-4214-8b19-e0e34a17f6b1 location:westus3] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:frontend-cert-prow-j7131520] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-l6h99n9b/pipeline@sha256:b84a27900c8a20ec255c2885caeb02d0d956d9ae792d8dcfba0223f93828f499 replicas:2 zoneCount:4] exitOnPanic:true mise:map[deploy:false policyLabel:ARM Policy] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-l6h99n9b/pipeline:pull workloadIdentityClientId:20c85050-271a-4f6d-b919-9e226971e33c workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:frontend workloadIdentityClientId:3e51afb0-72c4-4214-8b19-e0e34a17f6b1 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp] virtualService:map[host:rp.j7131520.hcpsvc.osadev.cloud]]" time=2026-03-19T20:08:41.333Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5xUzW7bMAy++ykEnxrAKrIEGLoMPfRn2GHoEKDDDh12kCXGYSuTBkUny9CHH5ykjVd4Qbvrp++HpCRmEdQIJG7FQzLnJjh16soIJ1zeg9cvSGGWVJCqwuygr66GZwhWQFskNc4/waMfmTHG5FdMC6xuXJMXJl8IkwIF67doBzlhu/RNXuzotyAr9NA7sU+it7FtDSro04DqotUlC/52ikxzjug3W1aMvLYu1EjWNWiVj0UfNdln/7eeWJdIQ/M5KsOkyBapEkhDjc8BpDMAUvRbhzdO7ha8gM6FVxhArqLbxTwrk99es3tIL2vZF8DhGpO0TZd92YYK9G03feE9t6T90AHyNTSRNzXQa93/FhxeKYjiohsVWIGFQFqCHG3wwsu8jfESKezv79msaWO05eHgRQmfncLabfoFVzvIwi8FIRePJu8HdMOEyvLKvr+jaOviwDdaDT/e7OfH7KEtQQgU0qfu46fs0ayXIGAUa0jq6saUoGsAMifBKXTwST4ZT97b8dS++/BtMp6Np7Pp5C4fmdNT80/O2Wx6dpePRtmjuWck84AUzpEIpLeumMxhS/XX08u9lD2aRrg7PRRamIbDjvw6k8JErgpTQ0qu6ixZAogpN73mAyT/JwAA//9EUhfPVAUAAA==" time=2026-03-19T20:08:41.335Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy time=2026-03-19T20:08:41.336Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress\ncontext deadline exceeded" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4004 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | e2e | failed | #4508 (merged) | 1 | e2e (1)pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:284]: Unexpected error:
<*errors.StatusError | 0xc000e04960>:
pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
Reason: "Forbidden",
Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:284]: Unexpected error:
<*errors.StatusError | 0xc000e04960>:
pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
Reason: "Forbidden",
Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4545 (closed) | 1 | provision (1)resource not ready, name: prometheus-operator, kind: Deployment, status: InProgress context deadline exceeded Show raw failuretime=2026-03-19T19:25:28.273Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-19T19:25:28.988Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:60cce7ea-f8ae-47ea-a2d1-1660743c2fc5]] prometheusSpec:map[externalLabels:map[cluster:prow-j0838272-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:61c6060c6bd283bf8633baf18e999c74ddb2cd65004784eb6a70204c6afa0e1f] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j0838272-svc-dcdl.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-5a84791bf5eb40b5a04b1a814789adaf/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-19T19:25:28.988Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-19T19:25:29.455Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-19T19:25:29.870Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-19T19:25:29.870Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-19T19:25:29.984Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-19T19:31:26.929Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="resource not ready, name: prometheus-operator, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-19T19:31:26.929Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-19T19:31:26.929Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-19T19:31:34.458Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: resource not ready, name: prometheus-operator, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:60cce7ea-f8ae-47ea-a2d1-1660743c2fc5]] prometheusSpec:map[externalLabels:map[cluster:prow-j0838272-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:61c6060c6bd283bf8633baf18e999c74ddb2cd65004784eb6a70204c6afa0e1f] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j0838272-svc-dcdl.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-5a84791bf5eb40b5a04b1a814789adaf/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-19T19:31:34.460Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yXz2/bOgzH7/4rhJwaIAraFK94zUMP773tNHQoumEDOuzASJyjxhIFim4WIH/84Lhp3B9pXTtX+suPKZGiqKxAUYyJSjaY1IWyICAwK/CIZrdo5JMLdpqEXchHqjZ9Bo8PJrzDsLGkCGZrHv7IlFJq8AX5zhn81xgqgwxGagBMcxO1p+CEWC/KGeokIKg9CjuTKlFk8ihzLNNgtA+002iKyCDE73d92eN/Cr9cfglxE7CHbWg6oYgLedJmo/C1ol7DKgn6B0JRJkG+pgJbL3qP7551vqluiu63oP/+v2vjP2AsaOWxV+IfQ1r/+7IUqFL1HWdzokWd0ZJBHIUnILDepXRv3/p/g8LZXoRGYv5zwbqQ96mEBqJ9Qbzo1NRekb2sI2lzLK72fn9dfV0+q2QokMVDgBx5zGWBXSD1IdSMBYFF7oLIMSBD0TmExd+p9h0bCgIuIGsTS10myFEnNBRs0kICxYHoHj3xShsw82cHvh+S06F24B6YlnV/PBxxSbxwIa/6sJ6t5GAp295/vXCRrKZlwD0d6XVK1QcOWZdPeIEsakZDXDWEftgWTftNCAcUTBpi7El4mFz6YZIQQ94p/w3IZgTQEF1CvutcB49plaXAZ2NLF1YHxqZu8Hcklh6N+hGlq39AqY5/R/fOsbeeOFpSXh2vGpdxvzntxVt93wKyn/9ku2L5WM3zKVur5RwZlTiPScBHNUNZIgZ1ZEGwMh8NJseTM318qk/Ov56cTyd/TSfnN4OhGo/VPs3pyXRydjMYDrO1uiUX1MIFe+FCQG68Qiio3eOj+ep4+tzI1ioyVV93gY5UJFuL20FGqqB8pDym6tbO1orYIqvZqrF4i8n8CQAA//8FiIteKw0AAA==" time=2026-03-19T19:31:34.461Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-19T19:31:34.461Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: prometheus-operator, kind: Deployment, status: InProgress\ncontext deadline exceeded" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #4544 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4543 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Show raw failuretime=2026-03-19T18:52:45.735Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-19T18:52:47.817Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-19T18:52:49.540Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=2504b525a81d38e7574160e2f0de8fb8c51a787cf186454a081120b8755f9cdb
time=2026-03-19T18:56:50.548Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4694272-svc/providers/Microsoft.Resources/deployments/2504b525a81d38e7574160e2f0de8fb8c51a787cf186454a081120b8755f9cdb/operationStatuses/08584276605168554876\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4694272-svc/providers/Microsoft.Resources/deployments/cluster-nkopotfn76bws\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4694272/providers/Microsoft.Resources/deployments/oidc-storage\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4694272/providers/Microsoft.Resources/deployments/storage\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4694272/providers/Microsoft.Resources/deployments/storage\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4694272/providers/Microsoft.Resources/deployments/oidcStorageRbac\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4694272/providers/Microsoft.Resources/deployments/oidcStorageRbac\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4234 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Show raw failuretime=2026-03-19T18:52:31.438Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-19T18:52:33.182Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-19T18:52:35.043Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3cc66decdbed55c45fdd4ee75868913bdbaa393a1029e6c0a3a6804a158f315f
time=2026-03-19T18:56:36.392Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0760832-svc/providers/Microsoft.Resources/deployments/3cc66decdbed55c45fdd4ee75868913bdbaa393a1029e6c0a3a6804a158f315f/operationStatuses/08584276605314679492\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0760832-svc/providers/Microsoft.Resources/deployments/cluster-ty5h7c4eu4lcw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4536 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Show raw failuretime=2026-03-19T18:54:20.685Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-03-19T18:54:21.984Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster
time=2026-03-19T18:54:23.201Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster deployment=0bfdc9c537f9456f74c5b80bcd861a7f9d2b569219c8aa1b927dbc89314df45f
time=2026-03-19T18:57:54.234Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8913152-mgmt-1/providers/Microsoft.Resources/deployments/0bfdc9c537f9456f74c5b80bcd861a7f9d2b569219c8aa1b927dbc89314df45f/operationStatuses/08584276604230221010\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8913152-mgmt-1/providers/Microsoft.Resources/deployments/cluster-w2b4uene52bv2\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8913152-mgmt-1/providers/Microsoft.Resources/deployments/prow-j8913152-mgmt-1-outbound-ip\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8913152-mgmt-1/providers/Microsoft.Resources/deployments/prow-j8913152-mgmt-1-outbound-ip\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4004 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Show raw failuretime=2026-03-19T18:46:17.431Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-19T18:46:20.647Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-19T18:46:22.202Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=80bb9318950571ddc624dbabe49f514c6d8f9c09f985a97c625bf27331d9acc0
time=2026-03-19T18:57:54.937Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4967424-svc/providers/Microsoft.Resources/deployments/80bb9318950571ddc624dbabe49f514c6d8f9c09f985a97c625bf27331d9acc0/operationStatuses/08584276609041060102\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4967424-svc/providers/Microsoft.Resources/deployments/cluster-sj245grfkcdbe\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/096555cd-32ef-5afa-acc2-c55ce48633c7\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/096555cd-32ef-5afa-acc2-c55ce48633c7\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/ad556013-895e-5a6b-820b-5cf5e4c3463b\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/ad556013-895e-5a6b-820b-5cf5e4c3463b\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4513 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #4457 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4133 (merged) | 1 | e2e (1)expected 1 nodes, found 4 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:156]: Expected success, but got an error:
<*errors.errorString | 0xc000113e10>:
expected 1 nodes, found 4
{
s: "expected 1 nodes, found 4",
}
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:156]: Expected success, but got an error:
<*errors.errorString | 0xc000113e10>:
expected 1 nodes, found 4
{
s: "expected 1 nodes, found 4",
} |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4508 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4305 (closed) | 2 | e2e (2)ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc000558990>:
GET https://rp.j7724544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"name": "6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"status": "Failed",
"startTime": "2026-03-19T17:55:14.004165795Z",
"endTime": "2026-03-19T18:00:44.464234913Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-k9pmhk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Thu, 19 Mar 2026 18:00:47 GMT",
],
"Content-Length": ["1395"],
"X-Envoy-Upstream-Service-Time": ["11"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"c0c30370-9a3a-4374-bfd7-67496bcec6a6",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0007801c0>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 55, 58, 53, 53, 58, 49, 52, 46, 48, 48, 52, 49, 54, 53, 55, 57, 53, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 56, 58, 48, 48, 58, 52, 52, 46, 52, 54, 52, 50, 51, 52, 57, 49, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 110, 101, 116, 119, 111, 114, 107, 32, 115, 101, 99, 117, 114, 105, 116, 121, 32, 103, 114, 111, 117, 112, 32, 39, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 39, 32, 100, 101, 116, 97, 105, 108, 115, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 114, 103, 45, 107, 57, 112, 109, 104, 107, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 78, 101, 116, 119, 111, 114, 107, 47, 110, 101, 116, 119, 111, 114, 107, 83, 101, 99, 117, 114, 105, 116, 121, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 92, 110, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc000558990>:
GET https://rp.j7724544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"name": "6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"status": "Failed",
"startTime": "2026-03-19T17:55:14.004165795Z",
"endTime": "2026-03-19T18:00:44.464234913Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-k9pmhk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Thu, 19 Mar 2026 18:00:47 GMT",
],
"Content-Length": ["1395"],
"X-Envoy-Upstream-Service-Time": ["11"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"c0c30370-9a3a-4374-bfd7-67496bcec6a6",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0007801c0>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 55, 58, 53, 53, 58, 49, 52, 46, 48, 48, 52, 49, 54, 53, 55, 57, 53, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 56, 58, 48, 48, 58, 52, 52, 46, 52, 54, 52, 50, 51, 52, 57, 49, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 110, 101, 116, 119, 111, 114, 107, 32, 115, 101, 99, 117, 114, 105, 116, 121, 32, 103, 114, 111, 117, 112, 32, 39, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 39, 32, 100, 101, 116, 97, 105, 108, 115, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 114, 103, 45, 107, 57, 112, 109, 104, 107, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 78, 101, 116, 119, 111, 114, 107, 47, 110, 101, 116, 119, 111, 114, 107, 83, 101, 99, 117, 114, 105, 116, 121, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 92, 110, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredInterrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4351 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4351 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | e2e | failed | #4351 (merged) | 3 | e2e (3)failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013a82a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xk2ks9" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000929200>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291e0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291c0>{
msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013a82a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xk2ks9" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000929200>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291e0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291c0>{
msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfailed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00168a138>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-l9pvkm" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000c09040>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09020>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09000>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00168a138>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-l9pvkm" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000c09040>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09020>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09000>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredUpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc000eaced0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tmg7sm" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-tmg7sm, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-tmg7sm\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-tmg7sm, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00066b770>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-tmg7sm",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc000eaced0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tmg7sm" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-tmg7sm, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-tmg7sm\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-tmg7sm, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00066b770>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-tmg7sm",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #4513 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4505 (merged) | 3 | e2e (3)ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
<*fmt.wrapError | 0xc000792440>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vrwqp2" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"name": "1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.755191638Z",
"endTime": "2026-03-19T17:49:13.219999333Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000792420>{
msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again afte...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
<*fmt.wrapError | 0xc000792440>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vrwqp2" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"name": "1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.755191638Z",
"endTime": "2026-03-19T17:49:13.219999333Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000792420>{
msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again afte...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredInterrupted by User Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:185]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:185]: Interrupted by User ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapError | 0xc000f663a0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bq8pz8" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"name": "09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.335034804Z",
"endTime": "2026-03-19T17:49:12.841643046Z",
"error": {
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f66380>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapError | 0xc000f663a0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bq8pz8" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"name": "09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.335034804Z",
"endTime": "2026-03-19T17:49:12.841643046Z",
"error": {
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f66380>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4542 (merged) | 1 | e2e (1)Deserializaion Error: no output from command Show raw failureparkunlock(...)
/usr/local/go/src/runtime/proc.go:466
runtime.semacquire1(0xc00104b6d8, 0x0, 0x1, 0x0, 0x19)
/usr/local/go/src/runtime/sema.go:192 +0x21d fp=0xc000bab5f8 sp=0xc000bab590 pc=0x2aaa3dd
sync.runtime_SemacquireWaitGroup(0xc00079f540?, 0x40?)
/usr/local/go/src/runtime/sema.go:114 +0x2e fp=0xc000bab630 sp=0xc000bab5f8 pc=0x2acf9ae
sync.(*WaitGroup).Wait(0xc00104b6d0)
/usr/local/go/src/sync/waitgroup.go:206 +0x85 fp=0xc000bab658 sp=0xc000bab630 pc=0x2af2e05
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc0008b4a20, {0x7fcd46f0ce60, 0xc001037ec0}, {{0xc0008a1450, 0x1, 0x1}, 0x34630b8a000, {0x782a6c2, 0x8}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:305 +0x273 fp=0xc000bab768 sp=0xc000bab658 pc=0x4ec41f3
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc0008b4a20, {0x7fcd46f0ce60, 0xc001037ec0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:224 +0x225 fp=0xc000bab888 sp=0xc000bab768 pc=0x4ec3a85
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7fcd46f0ce60?, 0xc001037ec0?})
<autogenerated>:1 +0x33 fp=0xc000bab8b0 sp=0xc000bab888 pc=0x4ed1c13
runtime.call16(0xc0012c4c00, 0xc0002c0fe0, 0x0, 0x0, 0x0, 0x10, 0xc000babb08)
/usr/local/go/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000bab8d0 sp=0xc000bab8b0 pc=0x2ad4e83
runtime.reflectcall(0x7289c80?, 0xc001037ec0?, 0x2?, 0x785af6d?, 0x0?, 0x12?, 0x6ed16a0?)
<autogenerated>:1 +0x36 fp=0xc000bab910 sp=0xc000bab8d0 pc=0x2add716
reflect.Value.call({0x6867740?, 0xc0002c0fe0?, 0xc0011bc5b0?}, {0x781f521, 0x4}, {0xc000d5df60, 0x1, 0x0?})
/usr/local/go/src/reflect/value.go:581 +0xcc6 fp=0xc000babe70 sp=0xc000bab910 pc=0x2b106e6
reflect.Value.Call({0x6867740?, 0xc0002c0fe0?, 0xc0011bc5b0?}, {0xc000d5df60?, 0xc000d5df28?, 0x2ace28e?})
/usr/local/go/src/reflect/value.go:365 +0xb9 fp=0xc000babee8 sp=0xc000babe70 pc=0x2b0f7d9
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x827b940?, 0xc001037ec0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000babf88 sp=0xc000babee8 pc=0x2f00ef0
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000babfe0 sp=0xc000babf88 pc=0x2f139bb
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000babfe8 sp=0xc000babfe0 pc=0x2ad6921
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 85
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 1289 gp=0xc001601dc0 m=nil [select]:
runtime.gopark(0xc000d67ab8?, 0x2?, 0xa0?, 0x22?, 0xc000d67aac?)
/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000d67938 sp=0xc000d67918 pc=0x2ace28e
runtime.selectgo(0xc000d67ab8, 0xc000d67aa8, 0xc0005d2a80?, 0x0, 0x329dd3c?, 0x1)
/usr/local/go/src/runtime/select.go:351 +0x8b7 fp=0xc000d67a78 sp=0xc000d67938 pc=0x2aa9137
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266038, 0xc000b89ea0}, 0x783ee71?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000d67ae8 sp=0xc000d67a78 pc=0x329d496
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x82aa0a0, {0x8266038, 0xc000b89ea0}, 0xc0016490a0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x732 fp=0xc000d67de8 sp=0xc000d67ae8 pc=0x36d4232
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster({0x8266038, 0xc000b89d60}, 0xc0016ca180, {0xc0009d4b60, 0x1c}, {0xc0015f0920, 0x1b}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x218 fp=0xc000d67ef0 sp=0xc000d67de8 pc=0x4eaf118
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:316 +0x94 fp=0xc000d67f78 sp=0xc000d67ef0 pc=0x4eb0f34
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc000d67fe0 sp=0xc000d67f78 pc=0x376b450
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d67fe8 sp=0xc000d67fe0 pc=0x2ad6921
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1332
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95
rax 0xca
rbx 0x0
rcx 0x2ad8743
rdx 0x0
rdi 0xb364af0
rsi 0x80
rbp 0x7fff66157880
rsp 0x7fff66157838
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0xc0000a0008
r13 0x1
r14 0xb362a00
r15 0x5
rip 0x2ad8741
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command
parkunlock(...)
/usr/local/go/src/runtime/proc.go:466
runtime.semacquire1(0xc00104b6d8, 0x0, 0x1, 0x0, 0x19)
/usr/local/go/src/runtime/sema.go:192 +0x21d fp=0xc000bab5f8 sp=0xc000bab590 pc=0x2aaa3dd
sync.runtime_SemacquireWaitGroup(0xc00079f540?, 0x40?)
/usr/local/go/src/runtime/sema.go:114 +0x2e fp=0xc000bab630 sp=0xc000bab5f8 pc=0x2acf9ae
sync.(*WaitGroup).Wait(0xc00104b6d0)
/usr/local/go/src/sync/waitgroup.go:206 +0x85 fp=0xc000bab658 sp=0xc000bab630 pc=0x2af2e05
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc0008b4a20, {0x7fcd46f0ce60, 0xc001037ec0}, {{0xc0008a1450, 0x1, 0x1}, 0x34630b8a000, {0x782a6c2, 0x8}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:305 +0x273 fp=0xc000bab768 sp=0xc000bab658 pc=0x4ec41f3
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc0008b4a20, {0x7fcd46f0ce60, 0xc001037ec0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:224 +0x225 fp=0xc000bab888 sp=0xc000bab768 pc=0x4ec3a85
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7fcd46f0ce60?, 0xc001037ec0?})
<autogenerated>:1 +0x33 fp=0xc000bab8b0 sp=0xc000bab888 pc=0x4ed1c13
runtime.call16(0xc0012c4c00, 0xc0002c0fe0, 0x0, 0x0, 0x0, 0x10, 0xc000babb08)
/usr/local/go/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000bab8d0 sp=0xc000bab8b0 pc=0x2ad4e83
runtime.reflectcall(0x7289c80?, 0xc001037ec0?, 0x2?, 0x785af6d?, 0x0?, 0x12?, 0x6ed16a0?)
<autogenerated>:1 +0x36 fp=0xc000bab910 sp=0xc000bab8d0 pc=0x2add716
reflect.Value.call({0x6867740?, 0xc0002c0fe0?, 0xc0011bc5b0?}, {0x781f521, 0x4}, {0xc000d5df60, 0x1, 0x0?})
/usr/local/go/src/reflect/value.go:581 +0xcc6 fp=0xc000babe70 sp=0xc000bab910 pc=0x2b106e6
reflect.Value.Call({0x6867740?, 0xc0002c0fe0?, 0xc0011bc5b0?}, {0xc000d5df60?, 0xc000d5df28?, 0x2ace28e?})
/usr/local/go/src/reflect/value.go:365 +0xb9 fp=0xc000babee8 sp=0xc000babe70 pc=0x2b0f7d9
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x827b940?, 0xc001037ec0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000babf88 sp=0xc000babee8 pc=0x2f00ef0
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000babfe0 sp=0xc000babf88 pc=0x2f139bb
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000babfe8 sp=0xc000babfe0 pc=0x2ad6921
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 85
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 1289 gp=0xc001601dc0 m=nil [select]:
runtime.gopark(0xc000d67ab8?, 0x2?, 0xa0?, 0x22?, 0xc000d67aac?)
/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000d67938 sp=0xc000d67918 pc=0x2ace28e
runtime.selectgo(0xc000d67ab8, 0xc000d67aa8, 0xc0005d2a80?, 0x0, 0x329dd3c?, 0x1)
/usr/local/go/src/runtime/select.go:351 +0x8b7 fp=0xc000d67a78 sp=0xc000d67938 pc=0x2aa9137
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266038, 0xc000b89ea0}, 0x783ee71?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000d67ae8 sp=0xc000d67a78 pc=0x329d496
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x82aa0a0, {0x8266038, 0xc000b89ea0}, 0xc0016490a0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x732 fp=0xc000d67de8 sp=0xc000d67ae8 pc=0x36d4232
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster({0x8266038, 0xc000b89d60}, 0xc0016ca180, {0xc0009d4b60, 0x1c}, {0xc0015f0920, 0x1b}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x218 fp=0xc000d67ef0 sp=0xc000d67de8 pc=0x4eaf118
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:316 +0x94 fp=0xc000d67f78 sp=0xc000d67ef0 pc=0x4eb0f34
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc000d67fe0 sp=0xc000d67f78 pc=0x376b450
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d67fe8 sp=0xc000d67fe0 pc=0x2ad6921
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1332
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95
rax 0xca
rbx 0x0
rcx 0x2ad8743
rdx 0x0
rdi 0xb364af0
rsi 0x80
rbp 0x7fff66157880
rsp 0x7fff66157838
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0xc0000a0008
r13 0x1
r14 0xb362a00
r15 0x5
rip 0x2ad8741
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4494 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4536 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4494 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4385 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Show raw failuretime=2026-03-19T16:02:24.737Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-19T16:02:25.307Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-19T16:02:26.298Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=e52fd2d5066e5f609d2338831504b035a21018a9b956bf0f907eb07f86757b0f
time=2026-03-19T16:02:56.497Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/e52fd2d5066e5f609d2338831504b035a21018a9b956bf0f907eb07f86757b0f/operationStatuses/08584276707397171591\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-backend-kv-access\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-ert2g2qezajzq\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-ert2g2qezajzq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-5rr4rbpkivz6y\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-5rr4rbpkivz6y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-5rr4rbpkivz6y\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-5rr4rbpkivz6y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-ert2g2qezajzq\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-ert2g2qezajzq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-iqtgb5ozb5jju\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-iqtgb5ozb5jju\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-iqtgb5ozb5jju\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-iqtgb5ozb5jju\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-refresher-msi-kv-access\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-heazf5wzdhzaw\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-heazf5wzdhzaw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-hyebjgkz4lbhe\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-hyebjgkz4lbhe\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-4m7g5555cws4s\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9550208-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-4m7g5555cws4s\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4508 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4541 (merged) | 1 | e2e (1)failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0011e6468>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bbz7dj" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000884cc0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884ca0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884c80>{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09b0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0011e6468>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bbz7dj" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000884cc0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884ca0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884c80>{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09b0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4393 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4532 (merged) | 1 | e2e (1)ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
<*fmt.wrapError | 0xc0007acfc0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-dwltsc" to finish creating: GET https://rp.j3457024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956",
"name": "e3935997-cedc-4c78-bded-528c72f0a956",
"status": "Failed",
"startTime": "2026-03-19T15:38:49.709273653Z",
"endTime": "2026-03-19T15:40:31.756262598Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dwltsc/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-dwltsc\" to finish creating: GET https://rp.j3457024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\",\n \"name\": \"e3935997-cedc-4c78-bded-528c72f0a956\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T15:38:49.709273653Z\",\n \"endTime\": \"2026-03-19T15:40:31.756262598Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dwltsc/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007acfa0>{
msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-dwltsc\" to finish creating: GET https://rp.j3457024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\",\n \"name\": \"e3935997-cedc-4c78-bded-528c72f0a956\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T15:38:49.709273653Z\",\n \"endTime\": \"2026-03-19T15:40:31.756262598Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dwltsc/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds a...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
<*fmt.wrapError | 0xc0007acfc0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-dwltsc" to finish creating: GET https://rp.j3457024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956",
"name": "e3935997-cedc-4c78-bded-528c72f0a956",
"status": "Failed",
"startTime": "2026-03-19T15:38:49.709273653Z",
"endTime": "2026-03-19T15:40:31.756262598Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dwltsc/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-dwltsc\" to finish creating: GET https://rp.j3457024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\",\n \"name\": \"e3935997-cedc-4c78-bded-528c72f0a956\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T15:38:49.709273653Z\",\n \"endTime\": \"2026-03-19T15:40:31.756262598Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dwltsc/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007acfa0>{
msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-dwltsc\" to finish creating: GET https://rp.j3457024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3935997-cedc-4c78-bded-528c72f0a956\",\n \"name\": \"e3935997-cedc-4c78-bded-528c72f0a956\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T15:38:49.709273653Z\",\n \"endTime\": \"2026-03-19T15:40:31.756262598Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dwltsc/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds a...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4045 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4540 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4536 (merged) | 1 | provision (1)Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": EOF Show raw failuretime=2026-03-19T15:16:39.998Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-19T15:16:40.694Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:4837a862-3da1-46eb-b760-0862e107baf6 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j1610624] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j1610624.hcpsvc.osadev.cloud] image:map[digest:sha256:fb9048ef756c29182c06d3eb373b01e676ff3afbc048e973a65ff2c158d74c73 registry:arohcpsvcdev.azurecr.io repository:ci-op-w1qwzszk/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-w1qwzszk/pipeline:pull workloadIdentityClientId:20990b84-9470-46ed-bace-0a875eb04b3b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:69bb32e9-27ee-4899-80e4-bff20dc9567d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-19T15:16:40.694Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T15:16:40.938Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate time=2026-03-19T15:16:41.252Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false time=2026-03-19T15:16:41.252Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T15:16:41.361Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T15:16:50.151Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" time=2026-03-19T15:16:50.151Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T15:16:50.151Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T15:16:51.973Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:4837a862-3da1-46eb-b760-0862e107baf6 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j1610624] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j1610624.hcpsvc.osadev.cloud] image:map[digest:sha256:fb9048ef756c29182c06d3eb373b01e676ff3afbc048e973a65ff2c158d74c73 registry:arohcpsvcdev.azurecr.io repository:ci-op-w1qwzszk/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-w1qwzszk/pipeline:pull workloadIdentityClientId:20990b84-9470-46ed-bace-0a875eb04b3b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:69bb32e9-27ee-4899-80e4-bff20dc9567d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-19T15:16:51.978Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yUzW7aQBDH736KkU8gsRakTaRS5UCSSpUqVYhwStXDej2YDesda2YNIuLhKwcIDgUH5eaZnf3tfPw9kcMAjEIVGxS4hUwHHXTqsEPpM5rwy/psKIGtz3uwdf3WBb65cIn+1SOlNnt3908EABA/Ii+twZExVPkQ9yAWFLHkcx3wP3N7Z0Lu+EgZ8oHJOeRPXVIOdYas0KEJlvwZyAOWjtYFXprp2XhlkIOdWVN/M84YZb5NXS9EWQmWlPU5o8ieNTI8rpy7sz6zPq9Dy8o5lR7sEwmMqjAnti+6LmpMzpr16yPO0UoVGNgaUfJxIa0cT2F+PoVHNIxhzLS0GfK903VJ75shpmwt/b6SQMVkp8EHnFlvj6YkSQOYaCY1N2WiXyrGxFBRR77RXCUB+ZQgdt+q0F7nuJ9bU0J3J9vdVFL7QC5jHKnxA+buH7pMkq2TPOr+ScDP6XQ8oerEA+/N6O/3aFGlyB4Dyo96CUi0gdUcGSHYAiXoooQUwwrRQyfTAWt3J77qX92o/hc1+DYdXA8HN8Ovg6e4C0kCLTHX/ae424028EzWw8L67NZ6j9xYXeThsLGaq+p4R0UbKJnq00OiPSgp2wZfBumBo7wHBYrovEYSZ8iQrhvFZyjmXwAAAP//Qqqu7WAFAAA=" time=2026-03-19T15:16:51.981Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T15:16:51.982Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4444 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4484 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4477 (closed) | 1 | e2e (1)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc000a96cc0>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-hv6bzb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-hv6bzb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea27c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc000a96cc0>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-hv6bzb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-hv6bzb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea27c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4425 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4305 (closed) | 2 | e2e (2)failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-<id>" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error:
<*fmt.wrapError | 0xc000e05cc0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-l922hd" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-l922hd\" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer",
err: <*fmt.wrapError | 0xc000e05c80>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-l922hd\" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer",
err: <*fmt.wrapError | 0xc000e05c60>{
msg: "failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-l922hd\" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer",
err: <http2.StreamError>{
StreamID: 25,
Code: 8,
Cause: <*errors.errorString | 0xb1bc5c0>{
s: "received from peer",
},
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error:
<*fmt.wrapError | 0xc000e05cc0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-l922hd" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-l922hd\" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer",
err: <*fmt.wrapError | 0xc000e05c80>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-l922hd\" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer",
err: <*fmt.wrapError | 0xc000e05c60>{
msg: "failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-l922hd\" at subscription scope to finish: stream error: stream ID 25; CANCEL; received from peer",
err: <http2.StreamError>{
StreamID: 25,
Code: 8,
Cause: <*errors.errorString | 0xb1bc5c0>{
s: "received from peer",
},
},
},
},
}
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:116]: Unexpected error:
<*fmt.wrapError | 0xc000b0fba0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-5fpjrx": PUT https://rp.j7560960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fpjrx\": PUT https://rp.j7560960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000b0fb80>{
msg: "failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fpjrx\": PUT https://rp.j7560960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0014f4840>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Thu, 19 Mar 2026 14:27:15 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["30"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"ed70a8b3-fa89-4990-a434-45df8d77753e",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000b0fa40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7560960.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
RawPath: "",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:116]: Unexpected error:
<*fmt.wrapError | 0xc000b0fba0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-5fpjrx": PUT https://rp.j7560960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fpjrx\": PUT https://rp.j7560960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000b0fb80>{
msg: "failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fpjrx\": PUT https://rp.j7560960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0014f4840>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Thu, 19 Mar 2026 14:27:15 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["30"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"ed70a8b3-fa89-4990-a434-45df8d77753e",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000b0fa40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7560960.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fpjrx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
RawPath: "",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4532 (merged) | 1 | provision (1)resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded Show raw failuretime=2026-03-19T13:47:01.252Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-19T13:47:01.909Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj6498688.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj6498688.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:6d9b1a79-b2f3-40b1-a69e-42423f2a74b9 databaseName:arohcpprow-rp-j6498688 databaseUrl:https://arohcpprow-rp-j6498688.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-h2tti4cs/pipeline@sha256:d4c0427893bf6e5995ed37f2b00c3119b0fe24b83fa546f811ad51c4c3d8a2f1 replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-h2tti4cs/pipeline:pull workloadIdentityClientId:052b85f6-11d4-4120-8d5c-2826936d6ac9 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:6d9b1a79-b2f3-40b1-a69e-42423f2a74b9 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-19T13:47:01.909Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-19T13:47:01.965Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-03-19T13:47:02.071Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-03-19T13:47:02.071Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-19T13:47:02.177Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-19T13:52:06.435Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-19T13:52:06.435Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-19T13:52:06.435Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-19T13:52:10.776Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj6498688.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj6498688.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:6d9b1a79-b2f3-40b1-a69e-42423f2a74b9 databaseName:arohcpprow-rp-j6498688 databaseUrl:https://arohcpprow-rp-j6498688.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-h2tti4cs/pipeline@sha256:d4c0427893bf6e5995ed37f2b00c3119b0fe24b83fa546f811ad51c4c3d8a2f1 replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-h2tti4cs/pipeline:pull workloadIdentityClientId:052b85f6-11d4-4120-8d5c-2826936d6ac9 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:6d9b1a79-b2f3-40b1-a69e-42423f2a74b9 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-19T13:52:10.779Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTwW7bPBCE73qKhU42IAaO8/8p6iKHNO2pSGE0PaXogSKnMmOKKywpGw788IXsuBYMufV1dr7hLrGbeSQSRG7FINIdWZ100qXHiMsXmPTFBTuLSVyoCtpLX3WNPxJWCDslNtoc5PGPjIgof+Dwy1WPuskLykttlghW6ddWoKQNydVQZmfp6lpYLUyTF39hz7q/sUcne2gLUfAwyXE44/zognWh6if/G7w3Mm+9H2Cb1ntVHvVTsE0LFvequ+A5e2c2O5f3vFY1kjgT1VvWAP4EWTmDXuVgPsAD0Cc0njc1QhrgBvxzQLo+EZIz+uQHLnjvCUaQ5sIrZyEPXsfY/6G4n0EtsVEr3fp0fs5HDi6xXNb2G3NvDLf7UQe82c8P2bItIQEJ8XO3sDHb0noBAXVLGJOuGyqR1kCgkdUJnTzKp5PprZrcqOv3369vZv+9m02mz/mYrq7onOf/6Wxy+5yPx9mWXtgFWrpg71wIkN6ZcaDjdfXP6vSesi01wl312GhBDdu9+bKQgjxXBdWIUVddJIuFULnpDW8Rze8AAAD///VVd6oMBAAA" time=2026-03-19T13:52:10.783Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-19T13:52:10.784Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4539 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4514 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4347 (merged) | 2 | e2e (2)CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc00104e5a0>:
failed to create NodePool oidc-wi-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool oidc-wi-np in resource group oidc-wi-49ztl8, error: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-49ztl8" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool oidc-wi-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool oidc-wi-np in resource group oidc-wi-49ztl8, error: failed waiting for nodepool=\"oidc-wi-np\" for cluster \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-49ztl8\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001405ee0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool oidc-wi-np in resource group oidc-wi-49ztl8",
},
<*fmt.wrapError | 0xc00093e1e0>{
msg: "failed waiting for nodepool=\"oidc-wi-np\" for cluster \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-49ztl8\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc00104e5a0>:
failed to create NodePool oidc-wi-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool oidc-wi-np in resource group oidc-wi-49ztl8, error: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-49ztl8" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool oidc-wi-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool oidc-wi-np in resource group oidc-wi-49ztl8, error: failed waiting for nodepool=\"oidc-wi-np\" for cluster \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-49ztl8\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001405ee0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool oidc-wi-np in resource group oidc-wi-49ztl8",
},
<*fmt.wrapError | 0xc00093e1e0>{
msg: "failed waiting for nodepool=\"oidc-wi-np\" for cluster \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-49ztl8\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredread access verification failed: get pod <pod> kube-system: pods is forbidden: User "system:sre-break-glass:<uuid>" cannot list resource "pods" in API group "" in the namespace "<namespace>" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.001s.
Expected success, but got an error:
<*fmt.wrapError | 0xc000b21780>:
read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
{
msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.joinError | 0xc000e6c2a0>{
errs: [
<*fmt.wrapError | 0xc000b21480>{
msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc00057a140>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
<*fmt.wrapError | 0xc000b21720>{
msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc00057bb80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.001s.
Expected success, but got an error:
<*fmt.wrapError | 0xc000b21780>:
read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
{
msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.joinError | 0xc000e6c2a0>{
errs: [
<*fmt.wrapError | 0xc000b21480>{
msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc00057a140>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
<*fmt.wrapError | 0xc000b21720>{
msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc00057bb80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
],
},
} |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4517 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4529 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4444 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4537 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4531 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4533 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4503 (merged) | 1 | e2e (1)Interrupted by User Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35]: Interrupted by User |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4481 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4535 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4517 (merged) | 1 | provision (1)requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve Show raw failuretime=2026-03-19T11:58:29.075Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-19T11:59:12.604Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n raise new_e\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n conn.connect()\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n self.sock = sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n cmd_result = self.invocation.execute(args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n raise ex\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n results.append(self._run_job(expanded_arg, cmd_copy))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n result = cmd_copy(params)\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n return self.handler(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n return op(**command_args)\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n login_server, _, password = get_login_credentials(\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n return _get_credentials(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 285, in _get_aad_token\n return _get_aad_token_after_challenge(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 159, in _get_aad_token_after_challenge\n response = requests.post(url=authhost, data=urlencode(content), headers=headers,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n return request(\"post\", url, data=data, json=json, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4347 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T11:38:52.390Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T11:38:55.084Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:38:55.084Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T11:38:55.084Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:38:55.084Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T11:38:55.084Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4403 (open) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code AnmPreconditionFailedException; detail message Error while processing network security perimeter association.; provider Microsoft.Network Show raw failuretime=2026-03-19T11:43:03.289Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-19T11:43:06.102Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-19T11:43:07.629Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=987cd1e56b36c548df3015be5c928fe6488455c5a3a725cd3294e876ba42e4e1
time=2026-03-19T11:55:40.655Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6649728-svc/providers/Microsoft.Resources/deployments/987cd1e56b36c548df3015be5c928fe6488455c5a3a725cd3294e876ba42e4e1/operationStatuses/08584276862987044363\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6649728-svc/providers/Microsoft.Resources/deployments/profile-begm52m5pz2do\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6649728-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j6649728-svc/resourceAssociations/nsp-j6649728-svc-ra-437d90f1-5adb-519e-85e4-35742f0071c8\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AnmPreconditionFailedException\\\",\\r\\n \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6649728-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j6649728-svc/resourceAssociations/nsp-j6649728-svc-ra-437d90f1-5adb-519e-85e4-35742f0071c8, Error message: There is already an operation in progress which requires exclusive lock on this service arohcpprow-rp-j6649728. Please retry the operation after sometime.\\\\r\\\\nActivityId: 37f4ab3a-4e0e-4a4a-8cd6-156f2d6abdc2, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4532 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T11:21:21.535Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T11:21:23.815Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:21:23.815Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T11:21:23.815Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:21:23.816Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T11:21:23.816Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4494 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4403 (open) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T10:34:51.543Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T10:34:54.096Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:34:54.096Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T10:34:54.097Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:34:54.097Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T10:34:54.097Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4503 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T10:24:05.189Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T10:24:07.308Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1107424639.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:24:07.309Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1107424639.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T10:24:07.309Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1107424639.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:24:07.309Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1107424639.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T10:24:07.309Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4505 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T10:33:04.131Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T10:33:06.290Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3601094097.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:33:06.290Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3601094097.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T10:33:06.290Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3601094097.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:33:06.290Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3601094097.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T10:33:06.290Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4529 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T10:25:42.372Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T10:25:44.551Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3146666533.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:25:44.551Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3146666533.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T10:25:44.551Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3146666533.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:25:44.551Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3146666533.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T10:25:44.551Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4533 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T10:23:22.006Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T10:23:24.390Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2262083953.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:23:24.390Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2262083953.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T10:23:24.390Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2262083953.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:23:24.390Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2262083953.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T10:23:24.390Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4503 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T09:59:52.254Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T09:59:54.613Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2667241276.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:59:54.613Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2667241276.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T09:59:54.613Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2667241276.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:59:54.613Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2667241276.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T09:59:54.613Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4514 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T09:37:43.299Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T09:37:46.061Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1700176026.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:37:46.061Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1700176026.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T09:37:46.061Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1700176026.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:37:46.062Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1700176026.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T09:37:46.062Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4532 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T09:49:35.382Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T09:49:38.018Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1162630994.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:49:38.018Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1162630994.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T09:49:38.018Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1162630994.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:49:38.018Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1162630994.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T09:49:38.018Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4517 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T09:47:01.978Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T09:47:06.249Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3597627977.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:47:06.249Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3597627977.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T09:47:06.250Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3597627977.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:47:06.250Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3597627977.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T09:47:06.250Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4514 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T09:03:32.103Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T09:03:34.838Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2996185070.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:03:34.838Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2996185070.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T09:03:34.838Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2996185070.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:03:34.838Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-2996185070.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T09:03:34.838Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4531 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T09:07:31.365Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T09:07:34.273Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1780113172.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:07:34.273Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1780113172.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T09:07:34.273Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1780113172.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T09:07:34.273Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1780113172.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T09:07:34.273Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4481 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T08:59:26.011Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T08:59:28.769Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3535488828.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T08:59:28.769Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3535488828.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T08:59:28.769Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3535488828.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T08:59:28.770Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3535488828.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T08:59:28.770Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4517 (merged) | 1 | provision (1)failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Show raw failuretime=2026-03-19T08:51:18.262Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T08:51:20.544Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1030451609.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T08:51:20.544Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1030451609.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T08:51:20.544Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1030451609.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T08:51:20.544Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1030451609.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T08:51:20.544Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4347 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4529 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | provision | failed | #4444 (merged) | 1 | provision (1)resource not ready, name: sessiongate, kind: Deployment, status: InProgress context deadline exceeded Show raw failuretime=2026-03-19T08:32:05.480Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-19T08:32:06.352Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:b3bd168a-18e0-4889-b4dd-953e3530d85b cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j6214400] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j6214400.hcpsvc.osadev.cloud] image:map[digest:sha256:796f9735ba4abe15568fe2c3d1460a6e33c0f57d0a05c556575de899c7724da1 registry:arohcpsvcdev.azurecr.io repository:ci-op-3fx1iyfr/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-3fx1iyfr/pipeline:pull workloadIdentityClientId:3d742a6c-79ed-4a6a-a366-734a1879a845 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:e5b3c84d-0aba-4357-b0df-7fbd92b15003 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-19T08:32:06.352Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T08:32:06.525Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate time=2026-03-19T08:32:06.777Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false time=2026-03-19T08:32:06.777Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T08:32:06.883Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T08:37:14.100Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="resource not ready, name: sessiongate, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-19T08:37:14.100Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T08:37:14.100Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T08:37:17.958Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: resource not ready, name: sessiongate, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:b3bd168a-18e0-4889-b4dd-953e3530d85b cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j6214400] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j6214400.hcpsvc.osadev.cloud] image:map[digest:sha256:796f9735ba4abe15568fe2c3d1460a6e33c0f57d0a05c556575de899c7724da1 registry:arohcpsvcdev.azurecr.io repository:ci-op-3fx1iyfr/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-3fx1iyfr/pipeline:pull workloadIdentityClientId:3d742a6c-79ed-4a6a-a366-734a1879a845 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:e5b3c84d-0aba-4357-b0df-7fbd92b15003 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-19T08:37:17.960Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5SUzW7bOhCF93oKQqsYEAUnuchtXWSRnwIFChSGm1WKLihqIjGhOOrMMEGCPHyhyK4FVbbTnX10+M2Qh5zEgygCxkgWWJ2r0ogRU3g4wuIerHx1oVywkAtVpnrpm2ngjwSPEN4Ubo3dyLMfiVJKpSv4FYHlIkoNQZw14jCkmUoZmB2Gygho6k3aRKlH39Ks53wHSyBLwkdXAl15wzymsG07yTywdiwOtQsVQefbIOjRWbiwFmOQv+pMlr3ykQVohX7s0OvfujHBVNBAj9wsvIbW4/NGPVxop19bIHF33cl1B3VHwDXQ3o1eWFpG7y9dKF2oOmsbvdfF9v9EA11CSO7lLaAlemef34p4j0/jMv8OaEDIWdZ8+CT2cgJKvXsPX25uliuME+jpcCMLNqv11b+GOxfc6HpyPliXG0Jd2zY3L5Egt9gME5+6IxaDEHrfBzbRwv5F2oMpgTR4sBPvZgi5nAx3yNof//sYo34OMNcP7kAYyc9PyUMsgAII8OdumHDyqp5qIFDiGmAxTasKkCeAoI5KI9DJR+nJ/ORMz0/18ceb+YfF6clifnabzlSeq52e/xfH/92ms1nyqu7RBfXgQnnuQgAajEAMajv5hiNvPOuSV9USdl+3jWaqxbI3vw+SKY9VphpgNlWHRCqBVPE82HwJbH8HAAD//5q3HKOoBQAA" time=2026-03-19T08:37:17.961Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-19T08:37:17.962Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: sessiongate, kind: Deployment, status: InProgress\ncontext deadline exceeded" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4528 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4517 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4497 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4396 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4425 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | e2e | failed | #4045 (open) | 1 | e2e (1)read access verification failed: get pod <pod> kube-system: pods is forbidden: User "system:sre-break-glass:<uuid>" cannot list resource "pods" in API group "" in the namespace "<namespace>" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.000s.
Expected success, but got an error:
<*fmt.wrapError | 0xc000fe61e0>:
read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
{
msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.joinError | 0xc000d315a8>{
errs: [
<*fmt.wrapError | 0xc000fc5e80>{
msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc000fd8be0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
<*fmt.wrapError | 0xc000fe6180>{
msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc000fd8e60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.000s.
Expected success, but got an error:
<*fmt.wrapError | 0xc000fe61e0>:
read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
{
msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.joinError | 0xc000d315a8>{
errs: [
<*fmt.wrapError | 0xc000fc5e80>{
msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc000fd8be0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
<*fmt.wrapError | 0xc000fe6180>{
msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc000fd8e60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
],
},
} |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4095 (open) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4396 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4444 (merged) | 1 | e2e (1)port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc0000fd940>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc000812ad0>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc0000fd940>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc000812ad0>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
},
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4517 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the request.; provider Microsoft.Network Show raw failuretime=2026-03-19T00:50:22.744Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-19T00:50:24.500Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-19T00:50:26.505Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=a5cab0b34a14fa55e3bc4dfe4c6c647490c0b66909a96c4f8622360e3cf9c5e4
time=2026-03-19T01:04:00.012Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2613760-svc/providers/Microsoft.Resources/deployments/a5cab0b34a14fa55e3bc4dfe4c6c647490c0b66909a96c4f8622360e3cf9c5e4/operationStatuses/08584277254600459045\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2613760-svc/providers/Microsoft.Resources/deployments/profile-kaxx74lugt6jc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2613760-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j2613760-svc/resourceAssociations/nsp-j2613760-svc-ra-34e5f7fa-afce-5359-977c-a16f66dca6ad\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2613760-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j2613760-svc/resourceAssociations/nsp-j2613760-svc-ra-34e5f7fa-afce-5359-977c-a16f66dca6ad, Error message: An error occurred while sending the request.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4473 (merged) | 0 | n/a |