Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 186 | 60.22% (off track) | -34.78pp | -11.52pp | 82.86% (145/175) | -4.93pp | 77.24% (112/145) | -10.18pp |
| INT | 95% - All E2E job runs | 45 | 46.67% (off track) | -48.33pp | +14.06pp | n/a | n/a | 46.67% (21/45) | +12.58pp |
| STG | 95% - All E2E job runs | 17 | 82.35% (off track) | -12.65pp | +22.35pp | n/a | n/a | 82.35% (14/17) | +15.69pp |
| PROD | 95% - All E2E job runs | 39 | 48.72% (off track) | -46.28pp | -18.78pp | n/a | n/a | 50.00% (19/38) | -21.05pp |
Environment: DEV
E2E Jobs
540
Success Rate
34.44%
Provision success
Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.72.39% (291/402)
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.63.92% (186/291)
E2E Jobs (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the number of runs in that DEV-only goal basis.186
Success Rate (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the overall success rate for that DEV-only goal basis.60.22%
Provision success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.82.86% (145/175)
E2E success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.77.24% (112/145)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
Successful runs (post-good + batches)
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 71.74% | 184 | rp-api-compat-all/parallel | Engineering should be able to retrieve kusto logs for a cluster and services | |
| 81.48% | 27 | rp-api-compat-all/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 88.54% | 288 | rp-api-compat-all/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 89.58% | 288 | rp-api-compat-all/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 90.88% | 285 | rp-api-compat-all/parallel | Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...full failure pattern: ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 38 | 7.04%38 of 540 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsMar 15: 0 · Mar 16: 62 · Mar 17: 11 · Mar 18: 6 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapError | 0xc0007aa280>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-b2rgrg" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
"name": "0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
"status": "Failed",
"startTime": "2026-03-18T15:53:21.008761573Z",
"endTime": "2026-03-18T15:54:00.900097609Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007aa260>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapError | 0xc0007aa280>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-b2rgrg" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
"name": "0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
"status": "Failed",
"startTime": "2026-03-18T15:53:21.008761573Z",
"endTime": "2026-03-18T15:54:00.900097609Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007aa260>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc0011f6d50>:
GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af",
"name": "c3df1fad-5b33-490c-8720-40caecee25af",
"status": "Failed",
"startTime": "2026-03-18T15:37:10.011477634Z",
"endTime": "2026-03-18T15:37:57.291321656Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"1f44e88a-3cf3-4e30-9689-672c60f0dee9",
],
"Date": [
"Wed, 18 Mar 2026 15:38:00 GMT",
],
"Content-Length": ["1561"],
"X-Envoy-Upstream-Service-Time": ["6"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0004c6c60>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 49, 48, 46, 48, 49, 49, 52, 55, 55, 54, 51, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 53, 55, 46, 50, 57, 49, 51, 50, 49, 54, 53, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 50, 51, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc0011f6d50>:
GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af",
"name": "c3df1fad-5b33-490c-8720-40caecee25af",
"status": "Failed",
"startTime": "2026-03-18T15:37:10.011477634Z",
"endTime": "2026-03-18T15:37:57.291321656Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"1f44e88a-3cf3-4e30-9689-672c60f0dee9",
],
"Date": [
"Wed, 18 Mar 2026 15:38:00 GMT",
],
"Content-Length": ["1561"],
"X-Envoy-Upstream-Service-Time": ["6"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0004c6c60>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 49, 48, 46, 48, 49, 49, 52, 55, 55, 54, 51, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 53, 55, 46, 50, 57, 49, 51, 50, 49, 54, 53, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 50, 51, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapError | 0xc00136e440>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sbghmb" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c",
"name": "7163cf65-26e8-4c55-8654-25f332c3301c",
"status": "Failed",
"startTime": "2026-03-18T15:39:30.642744115Z",
"endTime": "2026-03-18T15:40:26.468431346Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00136e420>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapError | 0xc00136e440>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sbghmb" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c",
"name": "7163cf65-26e8-4c55-8654-25f332c3301c",
"status": "Failed",
"startTime": "2026-03-18T15:39:30.642744115Z",
"endTime": "2026-03-18T15:40:26.468431346Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00136e420>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (22)
Affected runs (38) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 32 | 5.93%32 of 540 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsMar 15: 0 · Mar 16: 223 · Mar 17: 0 · Mar 18: 14 · Mar 19: 3 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013a82a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xk2ks9" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000929200>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291e0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291c0>{
msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013a82a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xk2ks9" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000929200>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291e0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0009291c0>{
msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-xk2ks9\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00168a138>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-l9pvkm" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000c09040>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09020>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09000>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00168a138>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-l9pvkm" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000c09040>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09020>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c09000>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-l9pvkm\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09c0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0011e6468>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bbz7dj" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000884cc0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884ca0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884c80>{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09b0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0011e6468>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bbz7dj" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000884cc0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884ca0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000884c80>{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bbz7dj\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb09b0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (23)
Affected runs (32) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Interrupted by Userfull failure pattern: Interrupted by User Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 7 | 1.30%7 of 540 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 3 · Mar 19: 3 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [:0]: Interrupted by User fail [:0]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:185]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:185]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35]: Interrupted by User Contributing tests (3)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | e2e | 6 | 1.11%6 of 540 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 1 · Mar 18: 1 · Mar 19: 2 · Mar 20: 1 · Mar 21: 0 · Mar 22: 0 | INT, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc0013b25a0>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tbqvdk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tbqvdk, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-tbqvdk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tbqvdk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013eb8c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tbqvdk",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc0013b25a0>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tbqvdk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tbqvdk, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-tbqvdk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tbqvdk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013eb8c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tbqvdk",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
<*fmt.wrapErrors | 0xc000faad50>:
failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-fkdqkt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded
{
msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<*fmt.wrapErrors | 0xc000faad20>{
msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
<*fmt.wrapErrors | 0xc000faad50>:
failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-fkdqkt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded
{
msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<*fmt.wrapErrors | 0xc000faad20>{
msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-fkdqkt\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001781280>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-fkdqkt",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc000a96cc0>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-hv6bzb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-hv6bzb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea27c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc000a96cc0>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-hv6bzb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-hv6bzb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea27c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hv6bzb",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (4)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]full failure pattern: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 35 | 6.48%35 of 540 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 14 · Mar 17: 1 · Mar 18: 20 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
<*errors.errorString | 0xc000da0f80>:
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
{
s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
}
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
<*errors.errorString | 0xc000da0f80>:
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
{
s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
}fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
<*errors.errorString | 0xc000a077d0>:
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
{
s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
}
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
<*errors.errorString | 0xc000a077d0>:
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
{
s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
}fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
<*errors.errorString | 0xc000511410>:
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
{
s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
}
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
<*errors.errorString | 0xc000511410>:
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
{
s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
}Contributing tests (1)
Affected runs (35) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.full failure pattern: ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | provision | 20 | 3.70%20 of 540 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 54 · Mar 17: 11 · Mar 18: 0 · Mar 19: 5 · Mar 20: 4 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-20T16:55:00.908Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-03-20T16:55:03.112Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster
time=2026-03-20T16:55:04.440Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster deployment=fb4a36dd0ff494fa35a9974bc8b6e8e5f0d46d48f18da1e495a99fed8805b844
time=2026-03-20T16:58:35.372Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0361344-mgmt-1/providers/Microsoft.Resources/deployments/fb4a36dd0ff494fa35a9974bc8b6e8e5f0d46d48f18da1e495a99fed8805b844/operationStatuses/08584275811817549292\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0361344-mgmt-1/providers/Microsoft.Resources/deployments/cluster-wtgam6mzv3hok\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0361344-mgmt-1/providers/Microsoft.Resources/deployments/prow-j0361344-mgmt-1-outbound-ip\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0361344-mgmt-1/providers/Microsoft.Resources/deployments/prow-j0361344-mgmt-1-outbound-ip\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-03-20T16:45:51.997Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-03-20T16:45:52.845Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster
time=2026-03-20T16:45:54.113Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster deployment=9d866b4507ed610ff4cd012d65ac8a88c2797ef7dec33752bfeb6b0d5ca19bcf
time=2026-03-20T16:55:56.289Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/9d866b4507ed610ff4cd012d65ac8a88c2797ef7dec33752bfeb6b0d5ca19bcf/operationStatuses/08584275817320700767\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/metrics-infra\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/1bfc0c3d-0803-54a4-98d0-9dd681a84c55\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/e87ba3a5-5c8b-591c-ae06-77192812ed2b\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/e8f77804-fe51-5e05-99f0-c3857de6fcd9\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/6417f1c9-492b-5383-89d9-d3a268821cc8\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/359981df-cf25-5199-bb86-db1b6021d9b5\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7762688-mgmt-1/providers/Microsoft.Resources/deployments/85f0b68f-9165-53f4-bad1-ddf2f1eaa33c\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-03-20T16:55:44.508Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-20T16:55:45.057Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-20T16:55:46.166Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=01e82a9dbd393631d4cdc403e6d2a13d0a5954cb6aa84fb7f837e5e0a6b958ce
time=2026-03-20T16:56:16.361Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/01e82a9dbd393631d4cdc403e6d2a13d0a5954cb6aa84fb7f837e5e0a6b958ce/operationStatuses/08584275811398670498\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-backend-kv-access\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-3sr6ewk4usq2w\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-3sr6ewk4usq2w\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-kzzwmv6egrcdg\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-kzzwmv6egrcdg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-kzzwmv6egrcdg\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-kzzwmv6egrcdg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-3sr6ewk4usq2w\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-3sr6ewk4usq2w\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-airyajwecijgk\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/cx-access-airyajwecijgk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-airyajwecijgk\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-airyajwecijgk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-refresher-msi-kv-access\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-tpqc7lbi73vw6\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-tpqc7lbi73vw6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-77z4miyuwcj5q\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-77z4miyuwcj5q\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-rskzz5hm4jrzk\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9695872-mgmt-1/providers/Microsoft.Resources/deployments/msi-access-rskzz5hm4jrzk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (5)
Affected runs (20) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...full failure pattern: ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | e2e | 17 | 3.15%17 of 540 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 4 · Mar 17: 2 · Mar 18: 10 · Mar 19: 4 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc000558990>:
GET https://rp.j7724544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"name": "6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"status": "Failed",
"startTime": "2026-03-19T17:55:14.004165795Z",
"endTime": "2026-03-19T18:00:44.464234913Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-k9pmhk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Thu, 19 Mar 2026 18:00:47 GMT",
],
"Content-Length": ["1395"],
"X-Envoy-Upstream-Service-Time": ["11"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"c0c30370-9a3a-4374-bfd7-67496bcec6a6",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0007801c0>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 55, 58, 53, 53, 58, 49, 52, 46, 48, 48, 52, 49, 54, 53, 55, 57, 53, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 56, 58, 48, 48, 58, 52, 52, 46, 52, 54, 52, 50, 51, 52, 57, 49, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 110, 101, 116, 119, 111, 114, 107, 32, 115, 101, 99, 117, 114, 105, 116, 121, 32, 103, 114, 111, 117, 112, 32, 39, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 39, 32, 100, 101, 116, 97, 105, 108, 115, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 114, 103, 45, 107, 57, 112, 109, 104, 107, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 78, 101, 116, 119, 111, 114, 107, 47, 110, 101, 116, 119, 111, 114, 107, 83, 101, 99, 117, 114, 105, 116, 121, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 92, 110, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc000558990>:
GET https://rp.j7724544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"name": "6e4be108-3630-4cd1-9bc4-5f9bf6c1d133",
"status": "Failed",
"startTime": "2026-03-19T17:55:14.004165795Z",
"endTime": "2026-03-19T18:00:44.464234913Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-k9pmhk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Thu, 19 Mar 2026 18:00:47 GMT",
],
"Content-Length": ["1395"],
"X-Envoy-Upstream-Service-Time": ["11"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"c0c30370-9a3a-4374-bfd7-67496bcec6a6",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0007801c0>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 54, 101, 52, 98, 101, 49, 48, 56, 45, 51, 54, 51, 48, 45, 52, 99, 100, 49, 45, 57, 98, 99, 52, 45, 53, 102, 57, 98, 102, 54, 99, 49, 100, 49, 51, 51, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 55, 58, 53, 53, 58, 49, 52, 46, 48, 48, 52, 49, 54, 53, 55, 57, 53, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 57, 84, 49, 56, 58, 48, 48, 58, 52, 52, 46, 52, 54, 52, 50, 51, 52, 57, 49, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 110, 101, 116, 119, 111, 114, 107, 32, 115, 101, 99, 117, 114, 105, 116, 121, 32, 103, 114, 111, 117, 112, 32, 39, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 39, 32, 100, 101, 116, 97, 105, 108, 115, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 114, 103, 45, 107, 57, 112, 109, 104, 107, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 78, 101, 116, 119, 111, 114, 107, 47, 110, 101, 116, 119, 111, 114, 107, 83, 101, 99, 117, 114, 105, 116, 121, 71, 114, 111, 117, 112, 115, 47, 99, 117, 115, 116, 111, 109, 101, 114, 45, 110, 115, 103, 45, 110, 97, 109, 101, 92, 110, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapError | 0xc000f663a0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bq8pz8" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"name": "09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.335034804Z",
"endTime": "2026-03-19T17:49:12.841643046Z",
"error": {
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f66380>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapError | 0xc000f663a0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bq8pz8" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"name": "09c2c3d3-9afa-4303-846a-0e7c745a8cbe",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.335034804Z",
"endTime": "2026-03-19T17:49:12.841643046Z",
"error": {
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f66380>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bq8pz8\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"name\": \"09c2c3d3-9afa-4303-846a-0e7c745a8cbe\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.335034804Z\",\n \"endTime\": \"2026-03-19T17:49:12.841643046Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-bq8pz8/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
<*fmt.wrapError | 0xc000792440>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vrwqp2" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"name": "1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.755191638Z",
"endTime": "2026-03-19T17:49:13.219999333Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000792420>{
msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again afte...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
<*fmt.wrapError | 0xc000792440>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vrwqp2" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"name": "1ef11d5a-18cb-48ed-afc1-00eea0d23317",
"status": "Failed",
"startTime": "2026-03-19T17:47:14.755191638Z",
"endTime": "2026-03-19T17:49:13.219999333Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000792420>{
msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-vrwqp2\" to finish creating: GET https://rp.j1746944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"name\": \"1ef11d5a-18cb-48ed-afc1-00eea0d23317\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-19T17:47:14.755191638Z\",\n \"endTime\": \"2026-03-19T17:49:13.219999333Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-vrwqp2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again afte...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (13)
Affected runs (17) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errorsfull failure pattern: failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | provision | 16 | 2.96%16 of 540 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 1 · Mar 18: 0 · Mar 19: 15 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-19T11:38:52.390Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T11:38:55.084Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:38:55.084Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T11:38:55.084Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:38:55.084Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-3640380727.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T11:38:55.084Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" time=2026-03-19T11:21:21.535Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T11:21:23.815Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:21:23.815Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T11:21:23.815Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T11:21:23.816Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1463722299.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T11:21:23.816Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" time=2026-03-19T10:34:51.543Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-19T10:34:54.096Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:34:54.096Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L20:32-L20:33 code=BCP019 time=2026-03-19T10:34:54.097Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location." time=2026-03-19T10:34:54.097Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-1706928777.bicepparam level=Error range=L21:31-L21:32 code=BCP019 time=2026-03-19T10:34:54.097Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors" Contributing tests (1)
Affected runs (16) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.full failure pattern: ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | e2e | 11 | 2.04%11 of 540 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 54 · Mar 17: 11 · Mar 18: 0 · Mar 19: 5 · Mar 20: 4 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error:
<*fmt.wrapError | 0xc0014d62c0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pp66nd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
},
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pp66nd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error:
<*fmt.wrapError | 0xc0014d62c0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pp66nd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
},
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pp66nd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error:
<*fmt.wrapError | 0xc000804c20>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-jjxpmq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000804c00>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/0...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error:
<*fmt.wrapError | 0xc000804c20>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-jjxpmq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000804c00>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/0...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error:
<*fmt.wrapError | 0xc00103a220>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-kx5x82-jnbcth" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00103a200>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list depl...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error:
<*fmt.wrapError | 0xc00103a220>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-kx5x82-jnbcth" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00103a200>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list depl...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (25)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' act...full failure pattern: ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 9 | 1.67%9 of 540 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 8 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0006754c0>:
failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster="cluster-zstream-4-19-wjcpfb" in resourcegroup="rg-zstream-upgrade-4-19-cct4x4" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120",
"name": "161f28e8-f27b-44a6-985c-acbfac3cb120",
"status": "Failed",
"startTime": "2026-03-18T15:48:29.77050876Z",
"endTime": "2026-03-18T15:49:46.080405554Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
},
{
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\",\n \"name\": \"161f28e8-f27b-44a6-985c-acbfac3cb120\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:48:29.77050876Z\",\n \"endTime\": \"2026-03-18T15:49:46.080405554Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0006754a0>{
msg: "failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfa...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0006754c0>:
failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster="cluster-zstream-4-19-wjcpfb" in resourcegroup="rg-zstream-upgrade-4-19-cct4x4" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120",
"name": "161f28e8-f27b-44a6-985c-acbfac3cb120",
"status": "Failed",
"startTime": "2026-03-18T15:48:29.77050876Z",
"endTime": "2026-03-18T15:49:46.080405554Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
},
{
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\",\n \"name\": \"161f28e8-f27b-44a6-985c-acbfac3cb120\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T15:48:29.77050876Z\",\n \"endTime\": \"2026-03-18T15:49:46.080405554Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0006754a0>{
msg: "failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfa...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000a0fee0>:
failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-tm8z9r" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8",
"name": "276ecb68-8f36-4788-a27a-b959fd469eb8",
"status": "Failed",
"startTime": "2026-03-16T14:30:45.201431526Z",
"endTime": "2026-03-16T14:31:35.067401014Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
},
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"name\": \"276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:30:45.201431526Z\",\n \"endTime\": \"2026-03-16T14:31:35.067401014Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a0fec0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n-----------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000a0fee0>:
failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-tm8z9r" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8",
"name": "276ecb68-8f36-4788-a27a-b959fd469eb8",
"status": "Failed",
"startTime": "2026-03-16T14:30:45.201431526Z",
"endTime": "2026-03-16T14:31:35.067401014Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
},
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"name\": \"276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:30:45.201431526Z\",\n \"endTime\": \"2026-03-16T14:31:35.067401014Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a0fec0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n-----------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error:
<*fmt.wrapError | 0xc000762460>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vd8tln" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b",
"name": "f7490142-2b7c-41f4-8adb-9053aa528f9b",
"status": "Failed",
"startTime": "2026-03-16T14:47:50.22465609Z",
"endTime": "2026-03-16T14:49:01.889796339Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
},
{
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"name\": \"f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:50.22465609Z\",\n \"endTime\": \"2026-03-16T14:49:01.889796339Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007622a0>{
msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESP...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error:
<*fmt.wrapError | 0xc000762460>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vd8tln" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b",
"name": "f7490142-2b7c-41f4-8adb-9053aa528f9b",
"status": "Failed",
"startTime": "2026-03-16T14:47:50.22465609Z",
"endTime": "2026-03-16T14:49:01.889796339Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
},
{
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"name\": \"f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:50.22465609Z\",\n \"endTime\": \"2026-03-16T14:49:01.889796339Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007622a0>{
msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESP...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (8)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
make[2]: *** [Makefile:32: deploy] Error 1full failure pattern: make[2]: *** [Makefile:32: deploy] Error 1 Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | provision | 8 | 1.48%8 of 540 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 8 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-16T19:59:43.936Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-16T20:00:56.397Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=a296ceb2-a43e-4725-9169-304a0c3e17b6 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=2e022157-7935-49d1-9846-1981212496c7 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj0348032.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj0348032.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j0348032 -n arohcp-prow-dbcs-j0348032 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0348032/providers/Microsoft.Network/dnszones/j0348032.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j0348032-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j0348032-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j0348032-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"95e11a6d-913e-4adb-9890-9e0ebd50bfa2\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j0348032-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0348032-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j0348032-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"time=2026-03-16T19:59:44.540Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-16T20:01:14.547Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=d47d6e1b-67d5-43b3-8321-cc3d6e757e42 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=a4766fe2-bcce-4945-9d82-6beb905a1e30 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3980288.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3980288.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3980288 -n arohcp-prow-dbcs-j3980288 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3980288/providers/Microsoft.Network/dnszones/j3980288.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3980288-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3980288-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j3980288-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"7fac78f7-d715-4063-b78f-ca3c8ffeb1ef\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3980288-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3980288-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3980288-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"time=2026-03-16T19:45:32.200Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-16T19:45:58.890Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=6ad615cf-74ad-45b9-98fd-af2a5733844d && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=f52e7edf-bae6-411f-bce8-6a372bd4b676 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj4297088.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj4297088.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j4297088 -n arohcp-prow-dbcs-j4297088 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4297088/providers/Microsoft.Network/dnszones/j4297088.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j4297088-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j4297088-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j4297088-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"b6de3e02-8f2f-4bf8-9489-ac97663eecaa\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j4297088-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4297088-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j4297088-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceledfull failure pattern: failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 7 | 1.30%7 of 540 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 9 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0010561e0>:
failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-hbjgwj" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0011132c0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0011132a0>{
msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0980c0>{
s: "context canceled",
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0010561e0>:
failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-hbjgwj" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0011132c0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0011132a0>{
msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0980c0>{
s: "context canceled",
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0007c6ab0>:
failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-n5jtqq" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0010cec40>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0010cec20>{
msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0980c0>{
s: "context canceled",
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0007c6ab0>:
failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-n5jtqq" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0010cec40>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0010cec20>{
msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0980c0>{
s: "context canceled",
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000b2e0d8>:
failed to cleanup resource group: failed waiting for resourcegroup="oidc-wi-rwm7g2" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0008a90e0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0008a90c0>{
msg: "failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0980c0>{
s: "context canceled",
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000b2e0d8>:
failed to cleanup resource group: failed waiting for resourcegroup="oidc-wi-rwm7g2" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0008a90e0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0008a90c0>{
msg: "failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0980c0>{
s: "context canceled",
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (5)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed post-install: Unauthorizedfull failure pattern: failed post-install: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | provision | 6 | 1.11%6 of 540 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 6 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-16T22:45:32.131Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-16T22:45:32.681Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-16T22:45:32.681Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:32.965Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-16T22:45:33.075Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-16T22:45:33.075Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:33.182Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:08.180Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: Unauthorized" time=2026-03-16T22:47:08.180Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:08.180Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:18.981Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-16T22:47:18.981Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:18.981Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized" time=2026-03-16T21:45:19.279Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T21:45:19.891Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:00d715af-ce8d-440f-8c1b-630fda34474f]] prometheusSpec:map[externalLabels:map[cluster:prow-j9755136-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8392ee99bb7b4c47be651b9c93097b71/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-ce8065f976124c569ca5653ccdbdf2a0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:45:19.891Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:45:20.256Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T21:45:20.682Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T21:45:20.682Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:45:20.833Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:28.992Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: Unauthorized" time=2026-03-16T21:47:28.992Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:28.992Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:45.274Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed post-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:00d715af-ce8d-440f-8c1b-630fda34474f]] prometheusSpec:map[externalLabels:map[cluster:prow-j9755136-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8392ee99bb7b4c47be651b9c93097b71/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-ce8065f976124c569ca5653ccdbdf2a0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:47:45.275Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yWTW/aTBDH7/4UK5+C5EUJz9M0osqhjXqqUlVp1UqpeljvTs0G7441MwYh8eErYwIuhAYZjjv7n9/OwLw4KUEUAWNNFljdKmfEiMlLuMD8Cax88tGNWcjHIlOt6bMJsDHBDOLKwpWxz+bBz0QppdK7smYBesAS0kylhnBiKx0wekHS0zoHzWIEdAAhb7kRpdmLvhVhAJlAzRorICNIB9QffHQ+Fmdw6mq/oLtv497XdE9r9cH7f6sf6r1sTQkkwURTAA2pLqEPZHrDre/QYhTjI5AOEJAW2ho7gfMiic8cI89NdV7iHGnqY6EZROcL6fejNvXbORcQgUzZ/0/a4UV0oAksUlOaJ2EpggDrTaOfhmFBMkWvoulAFiwQtKk8A81gr6360HowXuvNr0Azb6HT+0dNsWNAB6bTQdf31mId5fQYzkM4Ovz7Woz4WPyAfII4vcP42xc1GfEYd4edC555bX/2/25K704itA737RB5NXNtaxYMm2ZZ3dkV4uUE/8YHswExSBM2r71Dq2hfbAs2+fUu2Zbxx2aZcrJU8wkQKPGheT1UKgeZA0R14YxAY75IR5eja335n766/ja6Gv//Zjy6fEwHajhUBzVvx6Obx3QwSJbqCX1UUx/drY8RqPMJgFFtN3935e/u+mSpKsLmdhtopip0rfg4SKZKLDIVgNkUDRLJAal80UneAds/AQAA//+NZhXbqAgAAA==" time=2026-03-16T21:47:45.275Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:45.276Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized" time=2026-03-16T21:01:24.308Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T21:01:24.970Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:0c3e5218-5ebd-4cb6-aca0-9a4ce3c51e56]] prometheusSpec:map[externalLabels:map[cluster:prow-j6922880-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-a6c994f4b3b644e9a187298c90941783/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1cd0089dd64c44c8a8571eaf7bd7f61d/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:01:24.970Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:01:25.281Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T21:01:25.615Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T21:01:25.615Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:01:25.781Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:18.525Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: Unauthorized" time=2026-03-16T21:02:18.525Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:18.525Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:32.544Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed post-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:0c3e5218-5ebd-4cb6-aca0-9a4ce3c51e56]] prometheusSpec:map[externalLabels:map[cluster:prow-j6922880-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-a6c994f4b3b644e9a187298c90941783/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1cd0089dd64c44c8a8571eaf7bd7f61d/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:02:32.545Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6xWTW8TMRC976+wcmqkOGqCqKqgHqBwQkWoIJCKOHjtx8bN2mONZxsi5cejZPuxDYSmSY/79s2bZ489nqKGKEamhi2yOlPOiBFT1jii8hpWPvroJlnYx2qgWuiTCbiHcIO4RnIy9g7u/yiUUqr3mdwFRS/EvYHqJaYAmaLJm1+37HvksqnxmKNNDZZgoqnAQ25q7CMyO81t7NBSFOMjWNvU6CabCjrDUnRZC4mpX0g9IBAvtDV2ipeV5PxSO3ArmOcmHaSYyGmaR/xV7F1UmhK6810hgk29f6WbEhwhyNqkdKDC/f04TCYLsan2OgYdkUUWBG2Sz+CbvfY6koPG70QsB9ylRyr7xkfInHi2Z/j/vH8B33iLTvsxTFObdGiR9sRlMQIdIOztzkIdF5TA5h/trQ09r5ss4Etq3e9kYEvslpxPsreQ3vnofKye7eqbqb0z4mP1HeWUaHZO8ZevGjbiKW62bBd8zrf4xna+tZaaKIfXpSO0e13Wpi/admeCuUukM2S1tqztmhFaRutofe82kj8r63ukmhYBB636scjTuYufb4qH7vFh9VjnYqnmUzCU+IAsJiRVQuZAVEfOCFbwUW98PD7Rx6/06OTreDQ5Hk3Gr696fTUcqq2c8WR0etXr94uluiYf1cxHd+ZjBHdGDIrqYbLojhSbs0SxVIlp9ffB6EAlci15N5GBqqkaqIC8euKLpSJ2YFUuOot3yPZPAAAA//9TChsdCAkAAA==" time=2026-03-16T21:02:32.545Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:32.546Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized" Contributing tests (3)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
E2E Jobs
45
Success Rate
46.67%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.46.67% (21/45)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 77.50% | 40 | integration/parallel | Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle | |
| 84.44% | 45 | integration/parallel | Customer should create an HCP cluster and validate TLS certificates |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Deserializaion Error: no output from commandfull failure pattern: Deserializaion Error: no output from command Signal: NoiseSignal: Noise — source deserialization/no-output error | e2e | 1 | 2.22%1 of 45 job runs affected | NoiseSignal: Noise — source deserialization/no-output error | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||
Full failure examples (1)r.go:284
> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f55ac3c5a60, 0xc000636bd0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8278de0?, 0xc000636bd0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 97
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 09:18:44.139
"ts"="2026-03-18 09:18:44.139918" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-18 09:18:44.139996" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-52ql4t" "resourceGroup"="customer-rg-rfp52c"
"ts"="2026-03-18 09:18:44.346235" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-52ql4t" "resourceGroup"=""
"ts"="2026-03-18 09:18:45.945878" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-jwd5jl" "resourceGroup"="customer-rg-rfp52c"
"ts"="2026-03-18 09:18:46.101260" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-jwd5jl" "resourceGroup"=""
"ts"="2026-03-18 09:18:47.985220" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 09:18:47.985336" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 09:18:47.985396" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-18 09:18:47.986501" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-int-10"
"ts"="2026-03-18 09:18:47.986522" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-int-60"
"ts"="2026-03-18 09:18:47.986539" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-int-117"
"ts"="2026-03-18 09:18:47.986585" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-rfp52c"
"ts"="2026-03-18 09:18:48.234851" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-117" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 09:18:48.317393" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-03-18 09:18:48.362861" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-60" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 09:18:48.365533" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-117" "deployment"="msi-deployment-117"
"ts"="2026-03-18 09:18:48.465594" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="customer-infra-basic-hcp-cluster2-jwd5jl"
"ts"="2026-03-18 09:18:48.501722" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-60" "deployment"="msi-deployment-60"
"ts"="2026-03-18 09:18:48.625330" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-03-18 09:18:48.660006" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-10" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 09:18:48.790139" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="customer-infra-basic-hcp-cluster-52ql4t"
"ts"="2026-03-18 09:18:48.876784" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-10" "deployment"="msi-deployment-10"
"ts"="2026-03-18 09:18:49.020248" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-18 09:18:49.020387" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-18 09:18:49.020486" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-rfp52c"
Command Error: signal: killed
Deserializaion Error: no output from command
r.go:284
> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f55ac3c5a60, 0xc000636bd0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8278de0?, 0xc000636bd0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 97
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 09:18:44.139
"ts"="2026-03-18 09:18:44.139918" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-18 09:18:44.139996" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-52ql4t" "resourceGroup"="customer-rg-rfp52c"
"ts"="2026-03-18 09:18:44.346235" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-52ql4t" "resourceGroup"=""
"ts"="2026-03-18 09:18:45.945878" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-jwd5jl" "resourceGroup"="customer-rg-rfp52c"
"ts"="2026-03-18 09:18:46.101260" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-jwd5jl" "resourceGroup"=""
"ts"="2026-03-18 09:18:47.985220" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 09:18:47.985336" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 09:18:47.985396" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-18 09:18:47.986501" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-int-10"
"ts"="2026-03-18 09:18:47.986522" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-int-60"
"ts"="2026-03-18 09:18:47.986539" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-int-117"
"ts"="2026-03-18 09:18:47.986585" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-rfp52c"
"ts"="2026-03-18 09:18:48.234851" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-117" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 09:18:48.317393" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-03-18 09:18:48.362861" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-60" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 09:18:48.365533" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-117" "deployment"="msi-deployment-117"
"ts"="2026-03-18 09:18:48.465594" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="customer-infra-basic-hcp-cluster2-jwd5jl"
"ts"="2026-03-18 09:18:48.501722" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-60" "deployment"="msi-deployment-60"
"ts"="2026-03-18 09:18:48.625330" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-03-18 09:18:48.660006" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-10" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 09:18:48.790139" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-rfp52c" "deployment"="customer-infra-basic-hcp-cluster-52ql4t"
"ts"="2026-03-18 09:18:48.876784" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-int-10" "deployment"="msi-deployment-10"
"ts"="2026-03-18 09:18:49.020248" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-18 09:18:49.020387" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-18 09:18:49.020486" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-rfp52c"
Command Error: signal: killed
Deserializaion Error: no output from commandContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||
Value for field 'CommonName' failed to satisfy matcher.full failure pattern: Value for field 'CommonName' failed to satisfy matcher. Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 7 | 15.56%7 of 45 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 3 · Mar 17: 4 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
<string>: Microsoft TLS G2 RSA CA OCSP 02
to match regular expression
<string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
<string>: Microsoft TLS G2 RSA CA OCSP 02
to match regular expression
<string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
<string>: Microsoft TLS G2 RSA CA OCSP 04
to match regular expression
<string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
<string>: Microsoft TLS G2 RSA CA OCSP 04
to match regular expression
<string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+Contributing tests (1)
Affected runs (7)
| ||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Cannot request credential while credentials are being revoked; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Cannot request credential while credentials are being revoked; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | e2e | 5 | 11.11%5 of 45 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 1 · Mar 18: 0 · Mar 19: 0 · Mar 20: 1 · Mar 21: 2 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:249]: Unexpected error:
<*fmt.wrapError | 0xc000c14240>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while credentials are being revoked",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while credentials are being revoked\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0016c4840>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Retry-After": ["10"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Ms-Request-Id": [
"7f7535b0-0fe4-48cd-881d-3b603716e11e",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/5788f641-fa93-4db7-9a91-aa50926905b1",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Date": [
"Sat, 21 Mar 2026 18:38:33 GMT",
],
"Pragma": ["no-cache"],
"Content-Length": ["347"],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["41"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: E68D8F14163147F394B87CFEFB2E608F Ref B: CHI261140514060 Ref C: 2026-03-21T18:38:33Z",
],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Correlation-Request-Id": [
"18bf8c50-380a-4b05-815d-339e77cb569e",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260321T183833Z:18bf8c50-380a-4b05-815d-339e77cb569e",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Expires": ["-1"],
},
Body: <*exported.nopClosingBytesReader | 0xc000c14140>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:249]: Unexpected error:
<*fmt.wrapError | 0xc000c14240>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while credentials are being revoked",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while credentials are being revoked\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-j274qw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-pnlzgg\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0016c4840>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Retry-After": ["10"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Ms-Request-Id": [
"7f7535b0-0fe4-48cd-881d-3b603716e11e",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/5788f641-fa93-4db7-9a91-aa50926905b1",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Date": [
"Sat, 21 Mar 2026 18:38:33 GMT",
],
"Pragma": ["no-cache"],
"Content-Length": ["347"],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["41"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: E68D8F14163147F394B87CFEFB2E608F Ref B: CHI261140514060 Ref C: 2026-03-21T18:38:33Z",
],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Correlation-Request-Id": [
"18bf8c50-380a-4b05-815d-339e77cb569e",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260321T183833Z:18bf8c50-380a-4b05-815d-339e77cb569e",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Expires": ["-1"],
},
Body: <*exported.nopClosingBytesReader | 0xc000c14140>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:249]: Unexpected error:
<*fmt.wrapError | 0xc00059fc00>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while credentials are being revoked",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while credentials are being revoked\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e10750>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["347"],
"Retry-After": ["10"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/4dc1d6f1-e5cc-43f5-8c47-7eef96b2e0f5",
],
"X-Ms-Correlation-Request-Id": [
"cece0209-d710-4c31-bab2-21d80f15a352",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260321T050435Z:cece0209-d710-4c31-bab2-21d80f15a352",
],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Ms-Request-Id": [
"cad83257-e003-422e-acaf-729a0dee5f00",
],
"X-Envoy-Upstream-Service-Time": ["42"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: F1D21E1D4E7448A9A043A7A6C7CD9F5B Ref B: CHI261140511062 Ref C: 2026-03-21T05:04:35Z",
],
"Date": [
"Sat, 21 Mar 2026 05:04:34 GMT",
],
"Pragma": ["no-cache"],
},
Body: <*exported.nopClosingBytesReader | 0xc00059fae0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:249]: Unexpected error:
<*fmt.wrapError | 0xc00059fc00>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while credentials are being revoked",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while credentials are being revoked\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2kc5ht/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-79p5rk\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e10750>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["347"],
"Retry-After": ["10"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/4dc1d6f1-e5cc-43f5-8c47-7eef96b2e0f5",
],
"X-Ms-Correlation-Request-Id": [
"cece0209-d710-4c31-bab2-21d80f15a352",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260321T050435Z:cece0209-d710-4c31-bab2-21d80f15a352",
],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Ms-Request-Id": [
"cad83257-e003-422e-acaf-729a0dee5f00",
],
"X-Envoy-Upstream-Service-Time": ["42"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: F1D21E1D4E7448A9A043A7A6C7CD9F5B Ref B: CHI261140511062 Ref C: 2026-03-21T05:04:35Z",
],
"Date": [
"Sat, 21 Mar 2026 05:04:34 GMT",
],
"Pragma": ["no-cache"],
},
Body: <*exported.nopClosingBytesReader | 0xc00059fae0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:249]: Unexpected error:
<*fmt.wrapError | 0xc001328a20>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while credentials are being revoked",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while credentials are being revoked\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e088d0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260320T134725Z:e42c5286-5084-41f1-8a26-9a3abb7f5b29",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: AD1ED10A3FBE45A087E330C7FE2B7912 Ref B: CHI261140501029 Ref C: 2026-03-20T13:47:24Z",
],
"Date": [
"Fri, 20 Mar 2026 13:47:25 GMT",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"2d49c19c-df7f-4146-a1d8-66a45be53c4a",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"e42c5286-5084-41f1-8a26-9a3abb7f5b29",
],
"Content-Length": ["347"],
"Expires": ["-1"],
"X-Ms-Error-Code": ["Conflict"],
"X-Envoy-Upstream-Service-Time": ["60"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/698f44ec-1e20-4759-89d4-414a5c12bfb9",
],
"Content-Type": ["application/json"],
"Retry-After": ["10"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
},
Body: <*exported.nopClosingBytesReader | 0xc0013286e0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:249]: Unexpected error:
<*fmt.wrapError | 0xc001328a20>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while credentials are being revoked",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while credentials are being revoked\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-v9l57f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-hmbfpm\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e088d0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260320T134725Z:e42c5286-5084-41f1-8a26-9a3abb7f5b29",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: AD1ED10A3FBE45A087E330C7FE2B7912 Ref B: CHI261140501029 Ref C: 2026-03-20T13:47:24Z",
],
"Date": [
"Fri, 20 Mar 2026 13:47:25 GMT",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"2d49c19c-df7f-4146-a1d8-66a45be53c4a",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"e42c5286-5084-41f1-8a26-9a3abb7f5b29",
],
"Content-Length": ["347"],
"Expires": ["-1"],
"X-Ms-Error-Code": ["Conflict"],
"X-Envoy-Upstream-Service-Time": ["60"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/698f44ec-1e20-4759-89d4-414a5c12bfb9",
],
"Content-Type": ["application/json"],
"Retry-After": ["10"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
},
Body: <*exported.nopClosingBytesReader | 0xc0013286e0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||
not to be emptyfull failure pattern: not to be empty Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 5 | 11.11%5 of 45 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 2 · Mar 21: 3 · Mar 22: 0 | PROD, STG | ||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:126]: Expected
<[]*armredhatopenshifthcp.ImageDigestMirror | len:0, cap:0>: nil
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:126]: Expected
<[]*armredhatopenshifthcp.ImageDigestMirror | len:0, cap:0>: nil
not to be emptyContributing tests (1)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 4.44%2 of 45 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 1 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
<*fmt.wrapError | 0xc0010d4620>:
route host was never found: Get "https://agnhost-e2e-serving-app-bvxmn.apps.aro.f0b8a1v8r9h2u2b.os6f.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-bvxmn.apps.aro.f0b8a1v8r9h2u2b.os6f.uksouth.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc0004eb7d0>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-bvxmn.apps.aro.f0b8a1v8r9h2u2b.os6f.uksouth.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0xc0004eb7a0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 7, 24, 73, 129, 6, 189, 3, 123, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 49, 48, 52, 51, 52, 52, 53, 90, 23, 13, 50, 55, 48, 51, 50, 49, 48, 52, 51, 52, 52, 53, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 146, 154, 117, 19, 224, 169, 186, 58, 230, 251, 170, 128, 12, 92, 140, 43, 150, 232, 193, 85, 125, 186, 192, 131, 156, 99, 64, 239, 192, 84, 127, 85, 189, 114, 253, 212, 191, 60, 158, 147, 15, 141, 170, 204, 150, 241, 177, 59, 102, 231, 52, 19, 133, 224, 93, 111, 167, 251, 18, 180, 5, 60, 20, 189, 127, 178, 135, 78, 203, 93, 51, 114, 109, 53, 192, 55, 251, 146, 34, 89, 80, 242, 191, 212, 11, 251, 92, 82, 168, 202, 204, 236, 238, 218, 103, 249, 2, 3, 255, 188, 85, 122, 226, 237, 154, 87, 201, 92, 30, 248, 58, 219, 162, 210, 53, 157, 162, 250, 13, 20, 212, 154, 176, 116, 27, 254, 245, 181, 150, 10, 59, 176, 29, 71, 28, 57, 182, 49, 5, 158, 179, 170, 172, 231, 206, 41, 33, 116, 210, 92, 202, 59, 106, 58, 27, 239, 134, 65, 182, 26, 80, 0, 100, 43, 209, 139, 239, 124, 205, 146, 80, 246, 172, 110, 201, 17, 254, 203, 109, 163, 106, 211, 173, 4, 189, 234, 251, 98, 73, 158, 174, 247, 53, 180, 27, 93, 250, 50, 117, 20, 151, 173, 185, 134, 194, 240, 222, 51, 170, 47, 27, 54, 203, 225, 106, 136, 18, 33, 228, 201, 33, 155, 37, 143, 87, 148, 82, 254, 251, 98, 218, 65, 242, 232, 212, 33, 201, 225, 32, 190, 109, 194, 130, 84, 27, 198, 103, 231, 91, 141, 11, 103, 87, 81, 176, 213, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 249, 194, 155, 225, 108, 42, 222, 162, 76, 217, 84, 226, 207, 104, 26, 175, 56, 56, 202, 130, 109, 22, 8, 88, 98, 181, 203, 29, 149, 64, 217, 87, 135, 80, 81, 243, 101, 197, 164, 115, 41, 191, 99, 167, 60, 215, 132, 238, 202, 168, 104, 246, 147, 193, 228, 19, 132, 36, 200, 162, 230, 233, 91, 223, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 152, 169, 77, 135, 116, 37, 109, 16, 231, 51, 94, 199, 44, 186, 126, 209, 145, 141, 143, 17, 91, 108, 156, 142, 31, 143, 94, 218, 152, 163, 241, 6, 207, 199, 132, 180, 158, 186, 168, 185, 207, 94, 136, 94, 0, 61, 160, 112, 246, 162, 1, 251, 107, 226, 144, 122, 247, 227, 95, 232, 130, 0, 122, 143, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 102, 48, 98, 56, 97, 49, 118, 56, 114, 57, 104, 50, 117, 50, 98, 46, 111, 115, 54, 102, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 109, 149, 191, 123, 150, 135, 112, 137, 56, 3, 119, 61, 94, 253, 20, 213, 31, 8, 90, 58,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
<*fmt.wrapError | 0xc0010d4620>:
route host was never found: Get "https://agnhost-e2e-serving-app-bvxmn.apps.aro.f0b8a1v8r9h2u2b.os6f.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-bvxmn.apps.aro.f0b8a1v8r9h2u2b.os6f.uksouth.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc0004eb7d0>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-bvxmn.apps.aro.f0b8a1v8r9h2u2b.os6f.uksouth.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0xc0004eb7a0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 7, 24, 73, 129, 6, 189, 3, 123, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 49, 48, 52, 51, 52, 52, 53, 90, 23, 13, 50, 55, 48, 51, 50, 49, 48, 52, 51, 52, 52, 53, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 146, 154, 117, 19, 224, 169, 186, 58, 230, 251, 170, 128, 12, 92, 140, 43, 150, 232, 193, 85, 125, 186, 192, 131, 156, 99, 64, 239, 192, 84, 127, 85, 189, 114, 253, 212, 191, 60, 158, 147, 15, 141, 170, 204, 150, 241, 177, 59, 102, 231, 52, 19, 133, 224, 93, 111, 167, 251, 18, 180, 5, 60, 20, 189, 127, 178, 135, 78, 203, 93, 51, 114, 109, 53, 192, 55, 251, 146, 34, 89, 80, 242, 191, 212, 11, 251, 92, 82, 168, 202, 204, 236, 238, 218, 103, 249, 2, 3, 255, 188, 85, 122, 226, 237, 154, 87, 201, 92, 30, 248, 58, 219, 162, 210, 53, 157, 162, 250, 13, 20, 212, 154, 176, 116, 27, 254, 245, 181, 150, 10, 59, 176, 29, 71, 28, 57, 182, 49, 5, 158, 179, 170, 172, 231, 206, 41, 33, 116, 210, 92, 202, 59, 106, 58, 27, 239, 134, 65, 182, 26, 80, 0, 100, 43, 209, 139, 239, 124, 205, 146, 80, 246, 172, 110, 201, 17, 254, 203, 109, 163, 106, 211, 173, 4, 189, 234, 251, 98, 73, 158, 174, 247, 53, 180, 27, 93, 250, 50, 117, 20, 151, 173, 185, 134, 194, 240, 222, 51, 170, 47, 27, 54, 203, 225, 106, 136, 18, 33, 228, 201, 33, 155, 37, 143, 87, 148, 82, 254, 251, 98, 218, 65, 242, 232, 212, 33, 201, 225, 32, 190, 109, 194, 130, 84, 27, 198, 103, 231, 91, 141, 11, 103, 87, 81, 176, 213, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 249, 194, 155, 225, 108, 42, 222, 162, 76, 217, 84, 226, 207, 104, 26, 175, 56, 56, 202, 130, 109, 22, 8, 88, 98, 181, 203, 29, 149, 64, 217, 87, 135, 80, 81, 243, 101, 197, 164, 115, 41, 191, 99, 167, 60, 215, 132, 238, 202, 168, 104, 246, 147, 193, 228, 19, 132, 36, 200, 162, 230, 233, 91, 223, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 152, 169, 77, 135, 116, 37, 109, 16, 231, 51, 94, 199, 44, 186, 126, 209, 145, 141, 143, 17, 91, 108, 156, 142, 31, 143, 94, 218, 152, 163, 241, 6, 207, 199, 132, 180, 158, 186, 168, 185, 207, 94, 136, 94, 0, 61, 160, 112, 246, 162, 1, 251, 107, 226, 144, 122, 247, 227, 95, 232, 130, 0, 122, 143, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 102, 48, 98, 56, 97, 49, 118, 56, 114, 57, 104, 50, 117, 50, 98, 46, 111, 115, 54, 102, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 109, 149, 191, 123, 150, 135, 112, 137, 56, 3, 119, 61, 94, 253, 20, 213, 31, 8, 90, 58,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
<*fmt.wrapError | 0xc000e12c00>:
route host was never found: Get "https://agnhost-e2e-serving-app-vcfw6.apps.aro.q9c7h6e2w7p6c4m.f8k5.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-vcfw6.apps.aro.q9c7h6e2w7p6c4m.f8k5.uksouth.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc00092cc30>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-vcfw6.apps.aro.q9c7h6e2w7p6c4m.f8k5.uksouth.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0xc00092cc00>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 53, 224, 152, 96, 21, 167, 217, 201, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 49, 56, 50, 51, 50, 55, 53, 57, 90, 23, 13, 50, 55, 48, 51, 49, 56, 50, 51, 50, 55, 53, 57, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 137, 78, 205, 85, 199, 248, 179, 129, 134, 68, 36, 203, 162, 10, 128, 215, 1, 74, 91, 116, 62, 126, 190, 241, 168, 144, 110, 35, 152, 39, 215, 76, 22, 233, 158, 0, 42, 191, 24, 163, 76, 48, 4, 241, 70, 75, 61, 149, 161, 248, 157, 216, 21, 97, 6, 208, 183, 156, 81, 34, 80, 227, 244, 251, 169, 161, 34, 95, 149, 126, 166, 106, 95, 207, 0, 55, 48, 132, 207, 93, 64, 160, 113, 174, 79, 36, 191, 107, 48, 149, 188, 225, 46, 202, 49, 6, 115, 139, 174, 196, 98, 132, 244, 3, 95, 111, 27, 160, 244, 243, 101, 147, 126, 244, 41, 189, 115, 188, 237, 70, 244, 127, 34, 28, 10, 173, 143, 205, 149, 180, 184, 254, 16, 32, 197, 203, 14, 228, 97, 163, 198, 132, 241, 221, 157, 34, 154, 235, 73, 134, 229, 105, 125, 174, 102, 235, 223, 197, 12, 100, 163, 230, 125, 96, 197, 168, 218, 160, 196, 159, 201, 202, 210, 79, 77, 134, 201, 49, 105, 250, 24, 219, 64, 150, 144, 124, 154, 243, 147, 75, 141, 100, 225, 206, 150, 62, 86, 45, 93, 177, 56, 98, 23, 180, 61, 49, 84, 50, 239, 235, 175, 184, 173, 152, 119, 60, 156, 147, 81, 202, 133, 229, 88, 224, 134, 204, 105, 85, 128, 141, 211, 162, 111, 199, 22, 6, 13, 11, 194, 247, 30, 224, 113, 60, 122, 51, 213, 165, 253, 6, 36, 220, 151, 54, 143, 101, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 220, 49, 223, 226, 128, 113, 91, 117, 229, 53, 163, 156, 247, 135, 53, 234, 93, 79, 175, 247, 41, 93, 249, 185, 64, 107, 7, 233, 180, 254, 202, 192, 225, 52, 17, 190, 177, 193, 99, 101, 55, 231, 6, 26, 224, 3, 87, 191, 238, 145, 90, 12, 79, 227, 221, 18, 52, 131, 176, 103, 254, 114, 200, 86, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 96, 77, 79, 254, 38, 90, 139, 10, 37, 247, 206, 194, 82, 102, 238, 63, 128, 238, 166, 22, 251, 172, 206, 77, 204, 161, 50, 124, 66, 183, 196, 237, 163, 37, 121, 3, 202, 211, 226, 127, 10, 171, 226, 202, 104, 170, 99, 153, 62, 67, 212, 234, 206, 141, 144, 4, 35, 68, 228, 3, 69, 187, 249, 191, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 113, 57, 99, 55, 104, 54, 101, 50, 119, 55, 112, 54, 99, 52, 109, 46, 102, 56, 107, 53, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 41, 198, 145, 169, 222, 82, 247, 79, 220, 178, 90, 26, 252, 136, 89, 54, 145, 184, 38, 64, 219,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
<*fmt.wrapError | 0xc000e12c00>:
route host was never found: Get "https://agnhost-e2e-serving-app-vcfw6.apps.aro.q9c7h6e2w7p6c4m.f8k5.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-vcfw6.apps.aro.q9c7h6e2w7p6c4m.f8k5.uksouth.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc00092cc30>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-vcfw6.apps.aro.q9c7h6e2w7p6c4m.f8k5.uksouth.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0xc00092cc00>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 53, 224, 152, 96, 21, 167, 217, 201, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 49, 56, 50, 51, 50, 55, 53, 57, 90, 23, 13, 50, 55, 48, 51, 49, 56, 50, 51, 50, 55, 53, 57, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 137, 78, 205, 85, 199, 248, 179, 129, 134, 68, 36, 203, 162, 10, 128, 215, 1, 74, 91, 116, 62, 126, 190, 241, 168, 144, 110, 35, 152, 39, 215, 76, 22, 233, 158, 0, 42, 191, 24, 163, 76, 48, 4, 241, 70, 75, 61, 149, 161, 248, 157, 216, 21, 97, 6, 208, 183, 156, 81, 34, 80, 227, 244, 251, 169, 161, 34, 95, 149, 126, 166, 106, 95, 207, 0, 55, 48, 132, 207, 93, 64, 160, 113, 174, 79, 36, 191, 107, 48, 149, 188, 225, 46, 202, 49, 6, 115, 139, 174, 196, 98, 132, 244, 3, 95, 111, 27, 160, 244, 243, 101, 147, 126, 244, 41, 189, 115, 188, 237, 70, 244, 127, 34, 28, 10, 173, 143, 205, 149, 180, 184, 254, 16, 32, 197, 203, 14, 228, 97, 163, 198, 132, 241, 221, 157, 34, 154, 235, 73, 134, 229, 105, 125, 174, 102, 235, 223, 197, 12, 100, 163, 230, 125, 96, 197, 168, 218, 160, 196, 159, 201, 202, 210, 79, 77, 134, 201, 49, 105, 250, 24, 219, 64, 150, 144, 124, 154, 243, 147, 75, 141, 100, 225, 206, 150, 62, 86, 45, 93, 177, 56, 98, 23, 180, 61, 49, 84, 50, 239, 235, 175, 184, 173, 152, 119, 60, 156, 147, 81, 202, 133, 229, 88, 224, 134, 204, 105, 85, 128, 141, 211, 162, 111, 199, 22, 6, 13, 11, 194, 247, 30, 224, 113, 60, 122, 51, 213, 165, 253, 6, 36, 220, 151, 54, 143, 101, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 220, 49, 223, 226, 128, 113, 91, 117, 229, 53, 163, 156, 247, 135, 53, 234, 93, 79, 175, 247, 41, 93, 249, 185, 64, 107, 7, 233, 180, 254, 202, 192, 225, 52, 17, 190, 177, 193, 99, 101, 55, 231, 6, 26, 224, 3, 87, 191, 238, 145, 90, 12, 79, 227, 221, 18, 52, 131, 176, 103, 254, 114, 200, 86, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 96, 77, 79, 254, 38, 90, 139, 10, 37, 247, 206, 194, 82, 102, 238, 63, 128, 238, 166, 22, 251, 172, 206, 77, 204, 161, 50, 124, 66, 183, 196, 237, 163, 37, 121, 3, 202, 211, 226, 127, 10, 171, 226, 202, 104, 170, 99, 153, 62, 67, 212, 234, 206, 141, 144, 4, 35, 68, 228, 3, 69, 187, 249, 191, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 113, 57, 99, 55, 104, 54, 101, 50, 119, 55, 112, 54, 99, 52, 109, 46, 102, 56, 107, 53, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 41, 198, 145, 169, 222, 82, 247, 79, 220, 178, 90, 26, 252, 136, 89, 54, 145, 184, 38, 64, 219,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||
client rate limiter Wait returned an error: context deadline exceededfull failure pattern: client rate limiter Wait returned an error: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 4.44%2 of 45 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 1 · Mar 16: 2 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV | ||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 2 should fail after revocation, last error: client rate limiter Wait returned an error: context deadline exceeded
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 2 should fail after revocation, last error: client rate limiter Wait returned an error: context deadline exceeded
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 1 should fail after revocation, last error: client rate limiter Wait returned an error: context deadline exceeded
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 1 should fail after revocation, last error: client rate limiter Wait returned an error: context deadline exceeded
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 4.44%2 of 45 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:129]: Unexpected error:
<*fmt.wrapErrors | 0xc0010129f0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-lfq2nh, error: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-lfq2nh" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-lfq2nh, error: failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-lfq2nh\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0005ae7b0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-lfq2nh",
},
<*fmt.wrapError | 0xc0005b0500>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-lfq2nh\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:129]: Unexpected error:
<*fmt.wrapErrors | 0xc0010129f0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-lfq2nh, error: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-lfq2nh" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-lfq2nh, error: failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-lfq2nh\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0005ae7b0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-lfq2nh",
},
<*fmt.wrapError | 0xc0005b0500>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-lfq2nh\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:129]: Unexpected error:
<*fmt.wrapErrors | 0xc000a168d0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-d9mvv7, error: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-d9mvv7" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-d9mvv7, error: failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-d9mvv7\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00077f3b0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-d9mvv7",
},
<*fmt.wrapError | 0xc00047ad00>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-d9mvv7\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:129]: Unexpected error:
<*fmt.wrapErrors | 0xc000a168d0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-d9mvv7, error: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-d9mvv7" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-d9mvv7, error: failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-d9mvv7\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00077f3b0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-gpu-nodepool-NC4asT4v3-d9mvv7",
},
<*fmt.wrapError | 0xc00047ad00>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-d9mvv7\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||
RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.44%2 of 45 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 2 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc000ab30b0>:
failed waiting for hcpCluster="admin-cred-lifecycle-x7n2mn" in resourcegroup="admin-credential-lifecycle-test-j72vnp" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-x7n2mn in resource group admin-credential-lifecycle-test-j72vnp, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-x7n2mn\" in resourcegroup=\"admin-credential-lifecycle-test-j72vnp\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-x7n2mn in resource group admin-credential-lifecycle-test-j72vnp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b08d70>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-x7n2mn in resource group admin-credential-lifecycle-test-j72vnp",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc000ab30b0>:
failed waiting for hcpCluster="admin-cred-lifecycle-x7n2mn" in resourcegroup="admin-credential-lifecycle-test-j72vnp" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-x7n2mn in resource group admin-credential-lifecycle-test-j72vnp, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-x7n2mn\" in resourcegroup=\"admin-credential-lifecycle-test-j72vnp\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-x7n2mn in resource group admin-credential-lifecycle-test-j72vnp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b08d70>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-x7n2mn in resource group admin-credential-lifecycle-test-j72vnp",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc0004d1ef0>:
failed waiting for hcpCluster="admin-cred-lifecycle-hgqhmg" in resourcegroup="admin-credential-lifecycle-test-hsckzg" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-hgqhmg in resource group admin-credential-lifecycle-test-hsckzg, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-hgqhmg\" in resourcegroup=\"admin-credential-lifecycle-test-hsckzg\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-hgqhmg in resource group admin-credential-lifecycle-test-hsckzg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008a2580>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-hgqhmg in resource group admin-credential-lifecycle-test-hsckzg",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc0004d1ef0>:
failed waiting for hcpCluster="admin-cred-lifecycle-hgqhmg" in resourcegroup="admin-credential-lifecycle-test-hsckzg" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-hgqhmg in resource group admin-credential-lifecycle-test-hsckzg, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-hgqhmg\" in resourcegroup=\"admin-credential-lifecycle-test-hsckzg\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-hgqhmg in resource group admin-credential-lifecycle-test-hsckzg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008a2580>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-hgqhmg in resource group admin-credential-lifecycle-test-hsckzg",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.22%1 of 45 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 1 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:254]: New admin credentials should work after revocation
Expected success, but got an error:
<*errors.joinError | 0xc000d20db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0xc000d37080>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0xc000d37060>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0xc000c62be0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0xc000d83500>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0xc000d83360>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0xc0009d0280>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:254]: New admin credentials should work after revocation
Expected success, but got an error:
<*errors.joinError | 0xc000d20db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0xc000d37080>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0xc000d37060>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0xc000c62be0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0xc000d83500>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0xc000d83360>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0xc0009d0280>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||
found 1 managed resource groups left behind HCP clusters in cluster-listing-<id>full failure pattern: found 1 managed resource groups left behind HCP clusters in cluster-listing-<id> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.22%1 of 45 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 1 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc00111f080>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc
{
errs: [
<*errors.errorString | 0xc00135d4d0>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc00111f080>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc
{
errs: [
<*errors.errorString | 0xc00135d4d0>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 2.22%1 of 45 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 1 | DEV, PROD, STG | ||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapErrors | 0xc00052c2d0>:
failed to create HCP cluster cluster-zstream-4-19-pr8tbf, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: failed waiting for cluster="cluster-zstream-4-19-pr8tbf" in resourcegroup="rg-zstream-upgrade-4-19-8h6nq4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: context deadline exceeded
{
msg: "failed to create HCP cluster cluster-zstream-4-19-pr8tbf, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: failed waiting for cluster=\"cluster-zstream-4-19-pr8tbf\" in resourcegroup=\"rg-zstream-upgrade-4-19-8h6nq4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010a1340>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4",
},
<*fmt.wrapErrors | 0xc00052c270>{
msg: "failed waiting for cluster=\"cluster-zstream-4-19-pr8tbf\" in resourcegroup=\"rg-zstream-upgrade-4-19-8h6nq4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010a1340>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapErrors | 0xc00052c2d0>:
failed to create HCP cluster cluster-zstream-4-19-pr8tbf, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: failed waiting for cluster="cluster-zstream-4-19-pr8tbf" in resourcegroup="rg-zstream-upgrade-4-19-8h6nq4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: context deadline exceeded
{
msg: "failed to create HCP cluster cluster-zstream-4-19-pr8tbf, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: failed waiting for cluster=\"cluster-zstream-4-19-pr8tbf\" in resourcegroup=\"rg-zstream-upgrade-4-19-8h6nq4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010a1340>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4",
},
<*fmt.wrapErrors | 0xc00052c270>{
msg: "failed waiting for cluster=\"cluster-zstream-4-19-pr8tbf\" in resourcegroup=\"rg-zstream-upgrade-4-19-8h6nq4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010a1340>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pr8tbf in resource group rg-zstream-upgrade-4-19-8h6nq4",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||
Environment: STG
E2E Jobs
17
Success Rate
82.35%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.82.35% (14/17)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 87.50% | 16 | stage/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 87.50% | 16 | stage/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 | |
| 87.50% | 16 | stage/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 | |
| 87.50% | 16 | stage/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 | |
| 87.50% | 16 | stage/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 11.76%2 of 17 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 38 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapErrors | 0xc000cfd170>:
failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2bwhh8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: context deadline exceeded
{
msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2bwhh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00114ef20>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8",
},
<*fmt.wrapErrors | 0xc000cfd140>{
msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2bwhh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00114ef20>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapErrors | 0xc000cfd170>:
failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2bwhh8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: context deadline exceeded
{
msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2bwhh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00114ef20>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8",
},
<*fmt.wrapErrors | 0xc000cfd140>{
msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2bwhh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00114ef20>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2bwhh8",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc000f34570>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-kznm9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kznm9k, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-kznm9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kznm9k, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001035870>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kznm9k",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc000f34570>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-kznm9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kznm9k, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-kznm9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kznm9k, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001035870>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kznm9k",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc0014c40c0>:
failed to create HCP cluster ea-listlftzsc, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: failed waiting for cluster="ea-listlftzsc" in resourcegroup="ea-list-xrx8kk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-listlftzsc, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: failed waiting for cluster=\"ea-listlftzsc\" in resourcegroup=\"ea-list-xrx8kk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f66b50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk",
},
<*fmt.wrapErrors | 0xc0014c4090>{
msg: "failed waiting for cluster=\"ea-listlftzsc\" in resourcegroup=\"ea-list-xrx8kk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f66b50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc0014c40c0>:
failed to create HCP cluster ea-listlftzsc, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: failed waiting for cluster="ea-listlftzsc" in resourcegroup="ea-list-xrx8kk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-listlftzsc, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: failed waiting for cluster=\"ea-listlftzsc\" in resourcegroup=\"ea-list-xrx8kk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f66b50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk",
},
<*fmt.wrapErrors | 0xc0014c4090>{
msg: "failed waiting for cluster=\"ea-listlftzsc\" in resourcegroup=\"ea-list-xrx8kk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f66b50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listlftzsc in resource group ea-list-xrx8kk",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (19)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within 45 minutesfull failure pattern: Cluster should become ready within 45 minutes Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 11.76%2 of 17 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 2 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Interrupted by Userfull failure pattern: Interrupted by User Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 11.76%2 of 17 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 2 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
not to be emptyfull failure pattern: not to be empty Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.88%1 of 17 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 1 · Mar 22: 0 | INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:126]: Expected
<[]*armredhatopenshifthcp.ImageDigestMirror | len:0, cap:0>: nil
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:126]: Expected
<[]*armredhatopenshifthcp.ImageDigestMirror | len:0, cap:0>: nil
not to be emptyContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
E2E Jobs
39
Success Rate
48.72%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.50.00% (19/38)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 81.25% | 32 | prod/parallel | Customer should be able to list HCP clusters without node pools at both subscription and resource group levels | |
| 86.49% | 37 | prod/parallel | Customer should not be able to reuse subnets and NSGs between clusters | |
| 89.19% | 37 | prod/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 89.19% | 37 | prod/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 89.19% | 37 | prod/parallel | Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Deserializaion Error: no output from commandfull failure pattern: Deserializaion Error: no output from command Signal: NoiseSignal: Noise — source deserialization/no-output error | e2e | 2 | 5.13%2 of 39 job runs affected | NoiseSignal: Noise — source deserialization/no-output error | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 1 · Mar 19: 1 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)284
> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7fe3c01a6a60, 0xc00085a750})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc00085a750?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 99
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/19/26 04:21:45.455
"ts"="2026-03-19 04:21:45.455935" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-19 04:21:45.456006" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-qpcr94" "resourceGroup"="customer-rg-ggrxmh"
"ts"="2026-03-19 04:21:45.775140" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-qpcr94" "resourceGroup"=""
"ts"="2026-03-19 04:21:48.084018" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-lzd56b" "resourceGroup"="customer-rg-ggrxmh"
"ts"="2026-03-19 04:21:48.957488" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-lzd56b" "resourceGroup"=""
"ts"="2026-03-19 04:21:51.648746" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-19 04:21:51.648859" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-19 04:21:51.648920" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-19 04:21:51.650029" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-9"
"ts"="2026-03-19 04:21:51.650042" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-56"
"ts"="2026-03-19 04:21:51.650081" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-149"
"ts"="2026-03-19 04:21:51.650073" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-ggrxmh"
"ts"="2026-03-19 04:21:51.830715" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-56" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-19 04:21:51.980503" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-56" "deployment"="msi-deployment-56"
"ts"="2026-03-19 04:21:51.996787" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-03-19 04:21:52.175678" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-9" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-19 04:21:52.295989" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="customer-infra-basic-hcp-cluster2-lzd56b"
"ts"="2026-03-19 04:21:52.334633" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-9" "deployment"="msi-deployment-9"
"ts"="2026-03-19 04:21:52.554390" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-149" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-19 04:21:52.597003" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-03-19 04:21:52.680006" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-149" "deployment"="msi-deployment-149"
"ts"="2026-03-19 04:21:52.958897" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="customer-infra-basic-hcp-cluster-qpcr94"
"ts"="2026-03-19 04:21:53.267438" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-19 04:21:53.267585" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-19 04:21:53.267718" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-ggrxmh"
Command Error: signal: killed
Deserializaion Error: no output from command
284
> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7fe3c01a6a60, 0xc00085a750})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc00085a750?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 99
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/19/26 04:21:45.455
"ts"="2026-03-19 04:21:45.455935" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-19 04:21:45.456006" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-qpcr94" "resourceGroup"="customer-rg-ggrxmh"
"ts"="2026-03-19 04:21:45.775140" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-qpcr94" "resourceGroup"=""
"ts"="2026-03-19 04:21:48.084018" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-lzd56b" "resourceGroup"="customer-rg-ggrxmh"
"ts"="2026-03-19 04:21:48.957488" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-lzd56b" "resourceGroup"=""
"ts"="2026-03-19 04:21:51.648746" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-19 04:21:51.648859" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-19 04:21:51.648920" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-19 04:21:51.650029" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-9"
"ts"="2026-03-19 04:21:51.650042" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-56"
"ts"="2026-03-19 04:21:51.650081" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-149"
"ts"="2026-03-19 04:21:51.650073" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-ggrxmh"
"ts"="2026-03-19 04:21:51.830715" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-56" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-19 04:21:51.980503" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-56" "deployment"="msi-deployment-56"
"ts"="2026-03-19 04:21:51.996787" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-03-19 04:21:52.175678" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-9" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-19 04:21:52.295989" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="customer-infra-basic-hcp-cluster2-lzd56b"
"ts"="2026-03-19 04:21:52.334633" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-9" "deployment"="msi-deployment-9"
"ts"="2026-03-19 04:21:52.554390" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-149" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-19 04:21:52.597003" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-03-19 04:21:52.680006" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-149" "deployment"="msi-deployment-149"
"ts"="2026-03-19 04:21:52.958897" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-ggrxmh" "deployment"="customer-infra-basic-hcp-cluster-qpcr94"
"ts"="2026-03-19 04:21:53.267438" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-19 04:21:53.267585" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-19 04:21:53.267718" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-ggrxmh"
Command Error: signal: killed
Deserializaion Error: no output from command> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f54ec580e60, 0xc0009fb7d0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc0009fb7d0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 84
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 23:20:55.083
"ts"="2026-03-18 23:20:55.083532" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-18 23:20:55.083598" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "resourceGroup"="customer-rg-2hmq5s"
"ts"="2026-03-18 23:20:55.333961" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-kjjpjb" "resourceGroup"=""
"ts"="2026-03-18 23:20:56.369508" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "resourceGroup"="customer-rg-2hmq5s"
"ts"="2026-03-18 23:20:56.519635" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-wlxsb8" "resourceGroup"=""
"ts"="2026-03-18 23:20:57.115820" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 23:20:57.115955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 23:20:57.116022" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-18 23:20:57.117126" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-26"
"ts"="2026-03-18 23:20:57.117146" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-110"
"ts"="2026-03-18 23:20:57.117168" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-2hmq5s"
"ts"="2026-03-18 23:20:57.117208" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-36"
"ts"="2026-03-18 23:20:57.264846" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 23:20:57.327570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 23:20:57.396893" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="msi-deployment-110"
"ts"="2026-03-18 23:20:57.432057" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-03-18 23:20:57.548409" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="msi-deployment-36"
"ts"="2026-03-18 23:20:57.629848" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8"
"ts"="2026-03-18 23:20:57.773291" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 23:20:57.819000" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-03-18 23:20:57.904929" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster-kjjpjb"
"ts"="2026-03-18 23:20:58.268158" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="msi-deployment-26"
"ts"="2026-03-18 23:20:58.435912" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-18 23:20:58.436063" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-18 23:20:58.436192" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-2hmq5s"
Command Error: signal: killed
Deserializaion Error: no output from command
> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f54ec580e60, 0xc0009fb7d0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc0009fb7d0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 84
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 23:20:55.083
"ts"="2026-03-18 23:20:55.083532" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-18 23:20:55.083598" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "resourceGroup"="customer-rg-2hmq5s"
"ts"="2026-03-18 23:20:55.333961" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-kjjpjb" "resourceGroup"=""
"ts"="2026-03-18 23:20:56.369508" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "resourceGroup"="customer-rg-2hmq5s"
"ts"="2026-03-18 23:20:56.519635" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-wlxsb8" "resourceGroup"=""
"ts"="2026-03-18 23:20:57.115820" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 23:20:57.115955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-03-18 23:20:57.116022" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-18 23:20:57.117126" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-26"
"ts"="2026-03-18 23:20:57.117146" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-110"
"ts"="2026-03-18 23:20:57.117168" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-2hmq5s"
"ts"="2026-03-18 23:20:57.117208" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-36"
"ts"="2026-03-18 23:20:57.264846" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 23:20:57.327570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 23:20:57.396893" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="msi-deployment-110"
"ts"="2026-03-18 23:20:57.432057" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-03-18 23:20:57.548409" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="msi-deployment-36"
"ts"="2026-03-18 23:20:57.629848" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8"
"ts"="2026-03-18 23:20:57.773291" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-18 23:20:57.819000" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-03-18 23:20:57.904929" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster-kjjpjb"
"ts"="2026-03-18 23:20:58.268158" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="msi-deployment-26"
"ts"="2026-03-18 23:20:58.435912" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-18 23:20:58.436063" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-18 23:20:58.436192" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-2hmq5s"
Command Error: signal: killed
Deserializaion Error: no output from commandContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
or adjust the parameters in Gomega's 'format' package.full failure pattern: or adjust the parameters in Gomega's 'format' package. Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 6 | 15.38%6 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 6 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc0014e61e0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "ProviderError",
StatusCode: 502,
RawResponse: {
Status: "502 Bad Gateway",
StatusCode: 502,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T155939Z:81a9fbb3-a426-4d48-a5cc-b6e263995e30",
],
"X-Msedge-Ref": [
"Ref A: 59C38C75B1404B55A084E8EB38DC0136 Ref B: CHI261140501029 Ref C: 2026-03-19T15:58:38Z",
],
"Date": [
"Thu, 19 Mar 2026 15:59:39 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["162"],
"Content-Type": [
"application/json; charset=utf-8",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Request-Id": [
"81a9fbb3-a426-4d48-a5cc-b6e263995e30",
],
"X-Ms-Correlation-Request-Id": [
"81a9fbb3-a426-4d48-a5cc-b6e263995e30",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000c14000>{
s: "{\"error\":{\"code\":\"ProviderError\",\"message\":\"Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.\"}}",
i: 0,
},
ContentLength: 162,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "%24skiptoken=1Y%2FBaoQwGITf5T9vjLrZbhVKkWWhh0rLunvzEs2vBtckJNGWiu%2Fe0EPpK%2FQ2MwzMfCso%2FPSvUo0O8hXORXW9Ven5VrxDDoP3xuWUTlzxHidUPuJfs8Wo1RN1c%2BNaK42XWjnK4r3IBGYk2acNYdmRkYzxAzk0x8dGdDzJEqTG6kUKtI6WsrXa6c5HFxQv3L8ZVNUgO0%2BH1vya0312PtSfuZFkCSJMPaVxykj8QJKYGIuLxA%2FYdsDtVPS9xZ57FFc9ogr%2Fi0sJux%2FAktsRbYjW%2Bi9jDXn9Xyhr2GD7Bg%3D%3D&api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Authorization": ["redacted"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc0014e61e0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "ProviderError",
StatusCode: 502,
RawResponse: {
Status: "502 Bad Gateway",
StatusCode: 502,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T155939Z:81a9fbb3-a426-4d48-a5cc-b6e263995e30",
],
"X-Msedge-Ref": [
"Ref A: 59C38C75B1404B55A084E8EB38DC0136 Ref B: CHI261140501029 Ref C: 2026-03-19T15:58:38Z",
],
"Date": [
"Thu, 19 Mar 2026 15:59:39 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["162"],
"Content-Type": [
"application/json; charset=utf-8",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Request-Id": [
"81a9fbb3-a426-4d48-a5cc-b6e263995e30",
],
"X-Ms-Correlation-Request-Id": [
"81a9fbb3-a426-4d48-a5cc-b6e263995e30",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000c14000>{
s: "{\"error\":{\"code\":\"ProviderError\",\"message\":\"Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.\"}}",
i: 0,
},
ContentLength: 162,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "%24skiptoken=1Y%2FBaoQwGITf5T9vjLrZbhVKkWWhh0rLunvzEs2vBtckJNGWiu%2Fe0EPpK%2FQ2MwzMfCso%2FPSvUo0O8hXORXW9Ven5VrxDDoP3xuWUTlzxHidUPuJfs8Wo1RN1c%2BNaK42XWjnK4r3IBGYk2acNYdmRkYzxAzk0x8dGdDzJEqTG6kUKtI6WsrXa6c5HFxQv3L8ZVNUgO0%2BH1vya0312PtSfuZFkCSJMPaVxykj8QJKYGIuLxA%2FYdsDtVPS9xZ57FFc9ogr%2Fi0sJux%2FAktsRbYjW%2Bi9jDXn9Xyhr2GD7Bg%3D%3D&api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Authorization": ["redacted"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nilfail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc000c953e0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "ProviderError",
StatusCode: 502,
RawResponse: {
Status: "502 Bad Gateway",
StatusCode: 502,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 19 Mar 2026 10:24:04 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["162"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Request-Id": [
"a16a20c5-d011-43da-ae02-96abf758f533",
],
"X-Ms-Correlation-Request-Id": [
"a16a20c5-d011-43da-ae02-96abf758f533",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T102404Z:a16a20c5-d011-43da-ae02-96abf758f533",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: FD20AB1260094084B33CC10F073A5B77 Ref B: CHI261140514052 Ref C: 2026-03-19T10:23:58Z",
],
"Pragma": ["no-cache"],
},
Body: <*exported.nopClosingBytesReader | 0xc000f9a080>{
s: "{\"error\":{\"code\":\"ProviderError\",\"message\":\"Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.\"}}",
i: 0,
},
ContentLength: 162,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "%24skiptoken=1Y%2FBaoQwGITf5T9vjLrZbhVKkWWhh0rLunvzEs2vBtckJNGWiu%2Fe0EPpK%2FQ2MwzMfCso%2FPSvUo0O8hXORXW9Ven5VrxDDoP3xuWUTlzxHidUPuJfs8Wo1RN1c%2BNaK42XWjnK4r3IBGYk2acNYdmRkYzxAzk0x8dGdDzJEqTG6kUKtI6WsrXa6c5HFxQv3L8ZVNUgO0%2BH1vya0312PtSfuZFkCSJMPaVxykj8QJKYGIuLxA%2FYdsDtVPS9xZ57FFc9ogr%2Fi0sJux%2FAktsRbYjW%2Bi9jDXn9Xyhr2GD7Bg%3D%3D&api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc000c953e0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "ProviderError",
StatusCode: 502,
RawResponse: {
Status: "502 Bad Gateway",
StatusCode: 502,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 19 Mar 2026 10:24:04 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["162"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Request-Id": [
"a16a20c5-d011-43da-ae02-96abf758f533",
],
"X-Ms-Correlation-Request-Id": [
"a16a20c5-d011-43da-ae02-96abf758f533",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T102404Z:a16a20c5-d011-43da-ae02-96abf758f533",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: FD20AB1260094084B33CC10F073A5B77 Ref B: CHI261140514052 Ref C: 2026-03-19T10:23:58Z",
],
"Pragma": ["no-cache"],
},
Body: <*exported.nopClosingBytesReader | 0xc000f9a080>{
s: "{\"error\":{\"code\":\"ProviderError\",\"message\":\"Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.\"}}",
i: 0,
},
ContentLength: 162,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "%24skiptoken=1Y%2FBaoQwGITf5T9vjLrZbhVKkWWhh0rLunvzEs2vBtckJNGWiu%2Fe0EPpK%2FQ2MwzMfCso%2FPSvUo0O8hXORXW9Ven5VrxDDoP3xuWUTlzxHidUPuJfs8Wo1RN1c%2BNaK42XWjnK4r3IBGYk2acNYdmRkYzxAzk0x8dGdDzJEqTG6kUKtI6WsrXa6c5HFxQv3L8ZVNUgO0%2BH1vya0312PtSfuZFkCSJMPaVxykj8QJKYGIuLxA%2FYdsDtVPS9xZ57FFc9ogr%2Fi0sJux%2FAktsRbYjW%2Bi9jDXn9Xyhr2GD7Bg%3D%3D&api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nilfail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc000f8ea50>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "ProviderError",
StatusCode: 502,
RawResponse: {
Status: "502 Bad Gateway",
StatusCode: 502,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"d6d059ca-f5bf-439e-bbfd-46f05c53e078",
],
"X-Ms-Correlation-Request-Id": [
"d6d059ca-f5bf-439e-bbfd-46f05c53e078",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 43F7E9279933424097790033A2000E86 Ref B: CHI261140509034 Ref C: 2026-03-19T10:24:59Z",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["162"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T102559Z:d6d059ca-f5bf-439e-bbfd-46f05c53e078",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Date": [
"Thu, 19 Mar 2026 10:25:59 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0008f6380>{
s: "{\"error\":{\"code\":\"ProviderError\",\"message\":\"Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.\"}}",
i: 0,
},
ContentLength: 162,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "%24skiptoken=1Y%2FBaoQwGITf5T9vjLrZbhVKkWWhh0rLunvzEs2vBtckJNGWiu%2Fe0EPpK%2FQ2MwzMfCso%2FPSvUo0O8hXORXW9Ven5VrxDDoP3xuWUTlzxHidUPuJfs8Wo1RN1c%2BNaK42XWjnK4r3IBGYk2acNYdmRkYzxAzk0x8dGdDzJEqTG6kUKtI6WsrXa6c5HFxQv3L8ZVNUgO0%2BH1vya0312PtSfuZFkCSJMPaVxykj8QJKYGIuLxA%2FYdsDtVPS9xZ57FFc9ogr%2Fi0sJux%2FAktsRbYjW%2Bi9jDXn9Xyhr2GD7Bg%3D%3D&api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc000f8ea50>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "ProviderError",
StatusCode: 502,
RawResponse: {
Status: "502 Bad Gateway",
StatusCode: 502,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"d6d059ca-f5bf-439e-bbfd-46f05c53e078",
],
"X-Ms-Correlation-Request-Id": [
"d6d059ca-f5bf-439e-bbfd-46f05c53e078",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 43F7E9279933424097790033A2000E86 Ref B: CHI261140509034 Ref C: 2026-03-19T10:24:59Z",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["162"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T102559Z:d6d059ca-f5bf-439e-bbfd-46f05c53e078",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Date": [
"Thu, 19 Mar 2026 10:25:59 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0008f6380>{
s: "{\"error\":{\"code\":\"ProviderError\",\"message\":\"Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.\"}}",
i: 0,
},
ContentLength: 162,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "%24skiptoken=1Y%2FBaoQwGITf5T9vjLrZbhVKkWWhh0rLunvzEs2vBtckJNGWiu%2Fe0EPpK%2FQ2MwzMfCso%2FPSvUo0O8hXORXW9Ven5VrxDDoP3xuWUTlzxHidUPuJfs8Wo1RN1c%2BNaK42XWjnK4r3IBGYk2acNYdmRkYzxAzk0x8dGdDzJEqTG6kUKtI6WsrXa6c5HFxQv3L8ZVNUgO0%2BH1vya0312PtSfuZFkCSJMPaVxykj8QJKYGIuLxA%2FYdsDtVPS9xZ57FFc9ogr%2Fi0sJux%2FAktsRbYjW%2Bi9jDXn9Xyhr2GD7Bg%3D%3D&api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
ctx: <suppressed context>,
pat: nil,
matches: nil,
otherValues: nil,
},
TLS: {
Version: 772,
HandshakeComplete: true,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nilContributing tests (1)
Affected runs (6)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code InternalServerErrorfull failure pattern: ERROR CODE: DeploymentFailed; detail code InternalServerError Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 3 | 7.69%3 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 17 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:80]: Unexpected error:
<*fmt.wrapError | 0xc000a2bd60>:
failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-7sdmld" in resourcegroup="arm64-vm-cluster-bhd6ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a2bd40>{
msg: "failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008036e0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Correlation-Request-Id": [
"654f520d-e08b-4b13-9d40-7492126aea91",
],
"X-Content-Type-Options": ["nosniff"],
"Expires": ["-1"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:80]: Unexpected error:
<*fmt.wrapError | 0xc000a2bd60>:
failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-7sdmld" in resourcegroup="arm64-vm-cluster-bhd6ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a2bd40>{
msg: "failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008036e0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Correlation-Request-Id": [
"654f520d-e08b-4b13-9d40-7492126aea91",
],
"X-Content-Type-Options": ["nosniff"],
"Expires": ["-1"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: Unexpected error:
<*fmt.wrapError | 0xc000bc5580>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-t59jtk" in resourcegroup="rg-np-labels-taints-gt77vc" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bc5560>{
msg: "failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011eaea0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Request-Id": [
"af79b60e-5587-4117-9e37-523f09825369",
],
"Date": [
"Fri, 20 Mar 2026 09:44:49 GMT",
],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: Unexpected error:
<*fmt.wrapError | 0xc000bc5580>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-t59jtk" in resourcegroup="rg-np-labels-taints-gt77vc" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bc5560>{
msg: "failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011eaea0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Request-Id": [
"af79b60e-5587-4117-9e37-523f09825369",
],
"Date": [
"Fri, 20 Mar 2026 09:44:49 GMT",
],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error:
<*fmt.wrapError | 0xc000f82360>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-6fd9wv-x2gpnw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f82340>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f82320>{
msg: "failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error:
<*fmt.wrapError | 0xc000f82360>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-6fd9wv-x2gpnw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f82340>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f82320>{
msg: "failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (13)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 3 | 7.69%3 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 4 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapErrors | 0xc0009b8840>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-jmwp7b, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-jmwp7b" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-jmwp7b, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jmwp7b\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0005a92b0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-jmwp7b",
},
<*fmt.wrapError | 0xc001396680>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jmwp7b\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapErrors | 0xc0009b8840>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-jmwp7b, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-jmwp7b" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-jmwp7b, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jmwp7b\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0005a92b0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-jmwp7b",
},
<*fmt.wrapError | 0xc001396680>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jmwp7b\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:123]: Unexpected error:
<*fmt.wrapErrors | 0xc0013fc0c0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-k8q47m, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-k8q47m" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-k8q47m, error: failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-k8q47m\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00150edc0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-k8q47m",
},
<*fmt.wrapError | 0xc000c84460>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-k8q47m\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:123]: Unexpected error:
<*fmt.wrapErrors | 0xc0013fc0c0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-k8q47m, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-k8q47m" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-k8q47m, error: failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-k8q47m\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00150edc0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-k8q47m",
},
<*fmt.wrapError | 0xc000c84460>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-k8q47m\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapErrors | 0xc00092ae70>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-t7lnhv, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-t7lnhv" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-t7lnhv, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-t7lnhv\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001288490>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-t7lnhv",
},
<*fmt.wrapError | 0xc000960c40>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-t7lnhv\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapErrors | 0xc00092ae70>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-t7lnhv, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-t7lnhv" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-t7lnhv, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-t7lnhv\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001288490>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-t7lnhv",
},
<*fmt.wrapError | 0xc000960c40>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-t7lnhv\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredContributing tests (3)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-<id>full failure pattern: found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-<id> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 3 | 7.69%3 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 3 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc001a1ebd0>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-whz9b5
{
errs: [
<*errors.errorString | 0xc001598330>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-whz9b5",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc001a1ebd0>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-whz9b5
{
errs: [
<*errors.errorString | 0xc001598330>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-whz9b5",
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0012c3e30>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-dgq99r
{
errs: [
<*errors.errorString | 0xc0017cdab0>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-dgq99r",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0012c3e30>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-dgq99r
{
errs: [
<*errors.errorString | 0xc0017cdab0>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-dgq99r",
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0016cd758>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-6d7nns
{
errs: [
<*errors.errorString | 0xc000502840>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-6d7nns",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0016cd758>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-6d7nns
{
errs: [
<*errors.errorString | 0xc000502840>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-6d7nns",
},
],
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 5.13%2 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 7 · Mar 17: 0 · Mar 18: 11 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc000962220>:
failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster="cluster-zstream-4-20-hfk97l" in resourcegroup="rg-zstream-upgrade-4-20-6jjt78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49",
"name": "a535b9a7-79bf-4ae9-99b6-679319213e49",
"status": "Failed",
"startTime": "2026-03-18T22:10:00.515143027Z",
"endTime": "2026-03-18T22:11:58.502121438Z",
"error": {
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0009621e0>{
msg: "failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e803c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/a23e4d00-7c12-428e-b637-d6e87c428e63",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260318T221203Z:86a9a285-d268-4eaa-81f2-1b43b6865e4f",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["503"],
"X-Envoy-Upstream-Service-Time": ["10"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Correlation-Request-Id": [
"86a9a285-d268-4eaa-81f2-1b43b6865e4f",
],
"X-Msedge-Ref": [
"Ref A: 30110B94D9A9488595249B68A2A891EF Ref B: CHI261140505062 Ref C: 2026-03-18T22:12:03Z",
],
"Date": [
"Wed, 18 Mar 2026 22:12...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc000962220>:
failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster="cluster-zstream-4-20-hfk97l" in resourcegroup="rg-zstream-upgrade-4-20-6jjt78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49",
"name": "a535b9a7-79bf-4ae9-99b6-679319213e49",
"status": "Failed",
"startTime": "2026-03-18T22:10:00.515143027Z",
"endTime": "2026-03-18T22:11:58.502121438Z",
"error": {
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0009621e0>{
msg: "failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e803c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/a23e4d00-7c12-428e-b637-d6e87c428e63",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260318T221203Z:86a9a285-d268-4eaa-81f2-1b43b6865e4f",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["503"],
"X-Envoy-Upstream-Service-Time": ["10"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Correlation-Request-Id": [
"86a9a285-d268-4eaa-81f2-1b43b6865e4f",
],
"X-Msedge-Ref": [
"Ref A: 30110B94D9A9488595249B68A2A891EF Ref B: CHI261140505062 Ref C: 2026-03-18T22:12:03Z",
],
"Date": [
"Wed, 18 Mar 2026 22:12...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapError | 0xc000a8e6a0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-fncbll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b",
"name": "1dcfdab9-27df-483f-a009-42303866394b",
"status": "Failed",
"startTime": "2026-03-18T22:10:00.698917174Z",
"endTime": "2026-03-18T22:12:23.691383966Z",
"error": {
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a8e680>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00094b230>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["503"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"ee86427e-005d-4cc8-bd01-2b82a0056536",
],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
"CANADAEAST:20260318T221233Z:a23b9d02-f439-4d75-bcef-bec5f70b2688",
],
"X-Msedge-Ref": [
"Ref A: 2529055C24EC4D90ABA73CA7E8A5F278 Ref B: CHI261140504054 Ref C: 2026-03-18T22:12:33Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Envoy-Upstream-Service-Time": ["11"],
"X-Ms-Correlation-Request-Id": [
"a23b9d02-f439-4d75-bcef-bec5f70b2688",
],
"Strict-Transport-Security": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapError | 0xc000a8e6a0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-fncbll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b",
"name": "1dcfdab9-27df-483f-a009-42303866394b",
"status": "Failed",
"startTime": "2026-03-18T22:10:00.698917174Z",
"endTime": "2026-03-18T22:12:23.691383966Z",
"error": {
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a8e680>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00094b230>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["503"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"ee86427e-005d-4cc8-bd01-2b82a0056536",
],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
"CANADAEAST:20260318T221233Z:a23b9d02-f439-4d75-bcef-bec5f70b2688",
],
"X-Msedge-Ref": [
"Ref A: 2529055C24EC4D90ABA73CA7E8A5F278 Ref B: CHI261140504054 Ref C: 2026-03-18T22:12:33Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Envoy-Upstream-Service-Time": ["11"],
"X-Ms-Correlation-Request-Id": [
"a23b9d02-f439-4d75-bcef-bec5f70b2688",
],
"Strict-Transport-Security": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapError | 0xc000b943a0>:
failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster="ea-list8n6lb4" in resourcegroup="ea-list-gwfwm5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130",
"name": "b5aa74e8-43d5-4cff-b520-63b6b9163130",
"status": "Failed",
"startTime": "2026-03-18T21:58:12.305820102Z",
"endTime": "2026-03-18T21:59:35.853803496Z",
"error": {
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000b94380>{
msg: "failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011866c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260318T215943Z:3c196c67-a4c1-40eb-abc4-c5d8f5d91317",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 18 Mar 2026 21:59:43 GMT",
],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"e2863777-8b30-4013-8910-cf48bfc93c57",
],
"X-Envoy-Upstream-Service-Time": ["11"],
"X-Msedge-Ref": [
"Ref A: 694EB5D5CF094E458C2CC1B85768A697 Ref B: CHI261140510060 Ref C: 2026-03-18T21:59:43Z",
],
"Content-Length": ["503"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Expires...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapError | 0xc000b943a0>:
failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster="ea-list8n6lb4" in resourcegroup="ea-list-gwfwm5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130",
"name": "b5aa74e8-43d5-4cff-b520-63b6b9163130",
"status": "Failed",
"startTime": "2026-03-18T21:58:12.305820102Z",
"endTime": "2026-03-18T21:59:35.853803496Z",
"error": {
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000b94380>{
msg: "failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011866c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260318T215943Z:3c196c67-a4c1-40eb-abc4-c5d8f5d91317",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 18 Mar 2026 21:59:43 GMT",
],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"e2863777-8b30-4013-8910-cf48bfc93c57",
],
"X-Envoy-Upstream-Service-Time": ["11"],
"X-Msedge-Ref": [
"Ref A: 694EB5D5CF094E458C2CC1B85768A697 Ref B: CHI261140510060 Ref C: 2026-03-18T21:59:43Z",
],
"Content-Length": ["503"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Expires...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (13)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 5.13%2 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 9 · Mar 17: 0 · Mar 18: 7 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, INT, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
<*fmt.wrapErrors | 0xc000e6cb10>:
failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vm79bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded
{
msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011648c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
},
<*fmt.wrapErrors | 0xc000e6cae0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011648c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
<*fmt.wrapErrors | 0xc000e6cb10>:
failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vm79bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded
{
msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011648c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
},
<*fmt.wrapErrors | 0xc000e6cae0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011648c0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc00101e2d0>:
failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nxd528" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded
{
msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001374460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
},
<*fmt.wrapErrors | 0xc00101e2a0>{
msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001374460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc00101e2d0>:
failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nxd528" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded
{
msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001374460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
},
<*fmt.wrapErrors | 0xc00101e2a0>{
msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001374460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc00142ccf0>:
failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sjngpl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded
{
msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00158dc10>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
},
<*fmt.wrapErrors | 0xc00142ccc0>{
msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00158dc10>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc00142ccf0>:
failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sjngpl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded
{
msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00158dc10>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
},
<*fmt.wrapErrors | 0xc00142ccc0>{
msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00158dc10>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (11)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within 45 minutesfull failure pattern: Cluster should become ready within 45 minutes Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 5.13%2 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Interrupted by Userfull failure pattern: Interrupted by User Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 1 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:47]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:47]: Interrupted by User Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: ResourceGroupDeletionFailedfull failure pattern: ERROR CODE: ResourceGroupDeletionFailed Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 1 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000a84948>:
failed to cleanup resource group: failed waiting for resourcegroup="np-autoscaling-hsrbbh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJESFNSQkJILUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'np-autoscaling-hsrbbh' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0006c30a0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"np-autoscaling-hsrbbh\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJESFNSQkJILUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-hsrbbh' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0006c3080>{
msg: "failed waiting for resourcegroup=\"np-autoscaling-hsrbbh\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJESFNSQkJILUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-hsrbbh' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00092b0b0>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T034906Z:75f74363-b48d-43af-b448-6066e54e30d1",
],
"Content-Length": ["240"],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Request-Id": [
"75f74363-b48d-43af-b448-6066e54e30d1",
],
"Date": [
"Thu, 19 Mar 2026 03:49:05 GMT",
],
"Cache-Control": ["no-cache"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Pragma": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"75f74363-b48d-43af-b448-6066e54e30d1",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000a84948>:
failed to cleanup resource group: failed waiting for resourcegroup="np-autoscaling-hsrbbh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJESFNSQkJILUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'np-autoscaling-hsrbbh' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0006c30a0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"np-autoscaling-hsrbbh\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJESFNSQkJILUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-hsrbbh' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0006c3080>{
msg: "failed waiting for resourcegroup=\"np-autoscaling-hsrbbh\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJESFNSQkJILUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-hsrbbh' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00092b0b0>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260319T034906Z:75f74363-b48d-43af-b448-6066e54e30d1",
],
"Content-Length": ["240"],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Request-Id": [
"75f74363-b48d-43af-b448-6066e54e30d1",
],
"Date": [
"Thu, 19 Mar 2026 03:49:05 GMT",
],
"Cache-Control": ["no-cache"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Pragma": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"75f74363-b48d-43af-b448-6066e54e30d1",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: AuthorizationFailedfull failure pattern: ERROR CODE: AuthorizationFailed Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error:
<*fmt.wrapError | 0xc001504fc0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster-7hz8zk" in resourcegroup="customer-rg-clsh4k" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "AuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001504fa0>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0014bb1a0>{
ErrorCode: "AuthorizationFailed",
StatusCode: 403,
RawResponse: {
Status: "403 Forbidden",
StatusCode: 403,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Correlation-Request-Id": [
"06490acc-bbf6-4428-8144-e61db004ace3",
],
"X-Ms-Routing-Request-Id": [
"EASTUS2EUAP:20260316T060848Z:06490acc-bbf6-4428-8144-e61db004ace3",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Length": ["569"],
"Content-Type": [
"applic...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error:
<*fmt.wrapError | 0xc001504fc0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster-7hz8zk" in resourcegroup="customer-rg-clsh4k" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "AuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001504fa0>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0014bb1a0>{
ErrorCode: "AuthorizationFailed",
StatusCode: 403,
RawResponse: {
Status: "403 Forbidden",
StatusCode: 403,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Correlation-Request-Id": [
"06490acc-bbf6-4428-8144-e61db004ace3",
],
"X-Ms-Routing-Request-Id": [
"EASTUS2EUAP:20260316T060848Z:06490acc-bbf6-4428-8144-e61db004ace3",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Length": ["569"],
"Content-Type": [
"applic...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft....full failure pattern: ERROR CODE: AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapError | 0xc0008f2c20>:
failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-rkr4r8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "AuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008f2c00>{
msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0014046f0>{
ErrorCode: "AuthorizationFailed",
StatusCode: 403,
RawResponse: {
Status: "403 Forbidden",
StatusCode: 403,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["523"],
"X-Ms-Correlation-Request-Id": [
"2935921b-184b-4fc2-9c09-4c26121abf36",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Mon, 16 Mar 2026 05:50:31 GMT",
],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Request-Id": [
"2935921b-184b-4fc2-9c09-4c26121abf36",
],
"X-Ms-Routing-Request-Id": [
"EASTUS2EUAP:20260316T055032Z:2935921b-184b-4fc2-9c09-4c26121abf36",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapError | 0xc0008f2c20>:
failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-rkr4r8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "AuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008f2c00>{
msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0014046f0>{
ErrorCode: "AuthorizationFailed",
StatusCode: 403,
RawResponse: {
Status: "403 Forbidden",
StatusCode: 403,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["523"],
"X-Ms-Correlation-Request-Id": [
"2935921b-184b-4fc2-9c09-4c26121abf36",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Mon, 16 Mar 2026 05:50:31 GMT",
],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Request-Id": [
"2935921b-184b-4fc2-9c09-4c26121abf36",
],
"X-Ms-Routing-Request-Id": [
"EASTUS2EUAP:20260316T055032Z:2935921b-184b-4fc2-9c09-4c26121abf36",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization...full failure pattern: ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error:
<*fmt.wrapError | 0xc0004993c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-lqkgx2" in resourcegroup="external-auth-cluster-rhfph2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Forbidden",
"message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000499380>{
msg: "failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000baa6f0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error:
<*fmt.wrapError | 0xc0004993c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-lqkgx2" in resourcegroup="external-auth-cluster-rhfph2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Forbidden",
"message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000499380>{
msg: "failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000baa6f0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization...full failure pattern: ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error:
<*fmt.wrapError | 0xc00093d060>:
failed to create customer-infra: failed waiting for deployment "customer-infra-np-autoscale-cluster-5pp8qh" in resourcegroup="np-autoscaling-l7brcq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Forbidden",
"message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00093d020>{
msg: "failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008efc50>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: ED37EB9E1FA645FA954684A176851865 Ref B: CH1AA2020620045 Ref C: 2026-03-16T06:05:39Z",
],
"Date": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error:
<*fmt.wrapError | 0xc00093d060>:
failed to create customer-infra: failed waiting for deployment "customer-infra-np-autoscale-cluster-5pp8qh" in resourcegroup="np-autoscaling-l7brcq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Forbidden",
"message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00093d020>{
msg: "failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008efc50>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: ED37EB9E1FA645FA954684A176851865 Ref B: CH1AA2020620045 Ref C: 2026-03-16T06:05:39Z",
],
"Date": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization...full failure pattern: ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000760480>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-wk5cgh" in resourcegroup="rg-cluster-nsg-subnet-reuse-dvvbjq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Forbidden",
"message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000760460>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012bd110>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["839"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000760480>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-wk5cgh" in resourcegroup="rg-cluster-nsg-subnet-reuse-dvvbjq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Forbidden",
"message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000760460>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012bd110>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["839"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentityfull failure pattern: ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentity Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 1 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error:
<*fmt.wrapError | 0xc0016053a0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-9d7jkm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '92858e8c-aa82-4d12-af4e-bbaf740beedc', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5aedb7a4-fd70-4e59-8a0f-c36fcb99fe4a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6ad7864e-528a-4bd7-a91c-c7eead01227a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083025Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b474b493-2d00-4cd4-8b16-04c1849437da', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083024Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e178a66f-7a87-4e67-8562-6aafb6036181', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083023Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e59eefa1-b6f0-4d97-8d61-9506dc6c5ff4', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-9d7jkm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\\",\\r\\n \\\"message\\\": \\\"En...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error:
<*fmt.wrapError | 0xc0016053a0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-9d7jkm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '92858e8c-aa82-4d12-af4e-bbaf740beedc', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5aedb7a4-fd70-4e59-8a0f-c36fcb99fe4a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6ad7864e-528a-4bd7-a91c-c7eead01227a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083025Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b474b493-2d00-4cd4-8b16-04c1849437da', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083024Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e178a66f-7a87-4e67-8562-6aafb6036181', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083023Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e59eefa1-b6f0-4d97-8d61-9506dc6c5ff4', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-9d7jkm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\\",\\r\\n \\\"message\\\": \\\"En...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message ClientCertificateCredential: unable to resolve an endpoint: server response error: context dead...full failure pattern: ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message ClientCertificateCredential: unable to resolve an endpoint: server response error: context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
<*fmt.wrapError | 0xc001828480>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-99xb7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
"name": "6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
"status": "Failed",
"startTime": "2026-03-18T22:05:01.130113225Z",
"endTime": "2026-03-18T22:06:29.307118365Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001828460>{
msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerErro...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
<*fmt.wrapError | 0xc001828480>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-99xb7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
"name": "6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
"status": "Failed",
"startTime": "2026-03-18T22:05:01.130113225Z",
"endTime": "2026-03-18T22:06:29.307118365Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001828460>{
msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerErro...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.R...full failure pattern: ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 1 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
<*fmt.wrapError | 0xc00133a3c0>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-sv2zcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
"name": "b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
"status": "Failed",
"startTime": "2026-03-18T22:17:00.823479898Z",
"endTime": "2026-03-18T22:18:09.152905722Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
},
{
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00133a3a0>{
msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: Cl...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
<*fmt.wrapError | 0xc00133a3c0>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-sv2zcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
"name": "b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
"status": "Failed",
"startTime": "2026-03-18T22:17:00.823479898Z",
"endTime": "2026-03-18T22:18:09.152905722Z",
"error": {
"code": "MultipleErrorsOccurred",
"message": "Cluster provisioning failed due to multiple errors",
"details": [
{
"code": "InternalServerError",
"message": "insufficient public IP address quota: required 2, available 0"
},
{
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
},
{
"code": "InternalServerError",
"message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00133a3a0>{
msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: Cl...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
not to be emptyfull failure pattern: not to be empty Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 0 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 1 · Mar 22: 0 | INT, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:126]: Expected
<[]*armredhatopenshifthcp.ImageDigestMirror | len:0, cap:0>: nil
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:126]: Expected
<[]*armredhatopenshifthcp.ImageDigestMirror | len:0, cap:0>: nil
not to be emptyContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)full failure pattern: subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 15: 0 · Mar 16: 1 · Mar 17: 0 · Mar 18: 0 · Mar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.007s.
NFD operator should be installed successfully
Expected success, but got an error:
<*errors.errorString | 0xc001039e40>:
subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
{
s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.007s.
NFD operator should be installed successfully
Expected success, but got an error:
<*errors.errorString | 0xc001039e40>:
subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
{
s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||