Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 184 | 71.74% (off track) | -23.26pp | +9.30pp | 87.79% (151/172) | +19.56pp | 87.42% (132/151) | -6.48pp |
| INT | 95% - All E2E job runs | 46 | 32.61% (off track) | -62.39pp | -12.55pp | n/a | n/a | 34.09% (15/44) | -11.07pp |
| STG | 95% - All E2E job runs | 20 | 60.00% (off track) | -35.00pp | +16.25pp | n/a | n/a | 66.67% (12/18) | +22.92pp |
| PROD | 95% - All E2E job runs | 40 | 67.50% (off track) | -27.50pp | +21.67pp | n/a | n/a | 71.05% (27/38) | +25.22pp |
Environment: DEV
E2E Jobs
380
Success Rate
54.74%
Provision success
Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.82.50% (264/320)
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.78.79% (208/264)
E2E Jobs (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the number of runs in that DEV-only goal basis.184
Success Rate (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the overall success rate for that DEV-only goal basis.71.74%
Provision success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.87.79% (151/172)
E2E success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.87.42% (132/151)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
Successful runs (post-good + batches)
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 90.00% | 80 | rp-api-compat-all/parallel | Customer should be able to use workload identity via the cluster OIDC issuer URL |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...full failure pattern: ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 6 | 1.58%6 of 380 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 2 · Mar 11: 1 · Mar 12: 0 · Mar 13: 6 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc0013e27b0>:
GET https://rp.j4591104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1533905-6dac-43f4-90c6-d9eaae459507
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1533905-6dac-43f4-90c6-d9eaae459507",
"name": "d1533905-6dac-43f4-90c6-d9eaae459507",
"status": "Failed",
"startTime": "2026-03-13T09:37:46.323005448Z",
"endTime": "2026-03-13T09:38:20.435738147Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-28/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-28/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"2d4bb247-b8c7-41e4-8de7-8ac18e5b0f46",
],
"Date": [
"Fri, 13 Mar 2026 09:38:26 GMT",
],
"Content-Length": ["1539"],
"X-Envoy-Upstream-Service-Time": ["9"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc00063a2c0>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 100, 49, 53, 51, 51, 57, 48, 53, 45, 54, 100, 97, 99, 45, 52, 51, 102, 52, 45, 57, 48, 99, 54, 45, 100, 57, 101, 97, 97, 101, 52, 53, 57, 53, 48, 55, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 100, 49, 53, 51, 51, 57, 48, 53, 45, 54, 100, 97, 99, 45, 52, 51, 102, 52, 45, 57, 48, 99, 54, 45, 100, 57, 101, 97, 97, 101, 52, 53, 57, 53, 48, 55, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 51, 84, 48, 57, 58, 51, 55, 58, 52, 54, 46, 51, 50, 51, 48, 48, 53, 52, 52, 56, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 51, 84, 48, 57, 58, 51, 56, 58, 50, 48, 46, 52, 51, 53, 55, 51, 56, 49, 52, 55, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 56, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 107, 109, 115, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
<*exported.ResponseError | 0xc0013e27b0>:
GET https://rp.j4591104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1533905-6dac-43f4-90c6-d9eaae459507
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1533905-6dac-43f4-90c6-d9eaae459507",
"name": "d1533905-6dac-43f4-90c6-d9eaae459507",
"status": "Failed",
"startTime": "2026-03-13T09:37:46.323005448Z",
"endTime": "2026-03-13T09:38:20.435738147Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-28/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-28/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"2d4bb247-b8c7-41e4-8de7-8ac18e5b0f46",
],
"Date": [
"Fri, 13 Mar 2026 09:38:26 GMT",
],
"Content-Length": ["1539"],
"X-Envoy-Upstream-Service-Time": ["9"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc00063a2c0>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 100, 49, 53, 51, 51, 57, 48, 53, 45, 54, 100, 97, 99, 45, 52, 51, 102, 52, 45, 57, 48, 99, 54, 45, 100, 57, 101, 97, 97, 101, 52, 53, 57, 53, 48, 55, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 100, 49, 53, 51, 51, 57, 48, 53, 45, 54, 100, 97, 99, 45, 52, 51, 102, 52, 45, 57, 48, 99, 54, 45, 100, 57, 101, 97, 97, 101, 52, 53, 57, 53, 48, 55, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 51, 84, 48, 57, 58, 51, 55, 58, 52, 54, 46, 51, 50, 51, 48, 48, 53, 52, 52, 56, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 51, 84, 48, 57, 58, 51, 56, 58, 50, 48, 46, 52, 51, 53, 55, 51, 56, 49, 52, 55, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 56, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 107, 109, 115, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
<*fmt.wrapError | 0xc0008e25e0>:
failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rnjkhd" to finish creating: GET https://rp.j8910848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6",
"name": "0b86a247-0f2a-4a31-bc1f-16f9237d39d6",
"status": "Failed",
"startTime": "2026-03-13T01:28:43.185485913Z",
"endTime": "2026-03-13T01:30:19.825677573Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rnjkhd\" to finish creating: GET https://rp.j8910848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"name\": \"0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:43.185485913Z\",\n \"endTime\": \"2026-03-13T01:30:19.825677573Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008e25c0>{
msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rnjkhd\" to finish creating: GET https://rp.j8910848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"name\": \"0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:43.185485913Z\",\n \"endTime\": \"2026-03-13T01:30:19.825677573Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
<*fmt.wrapError | 0xc0008e25e0>:
failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rnjkhd" to finish creating: GET https://rp.j8910848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6",
"name": "0b86a247-0f2a-4a31-bc1f-16f9237d39d6",
"status": "Failed",
"startTime": "2026-03-13T01:28:43.185485913Z",
"endTime": "2026-03-13T01:30:19.825677573Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rnjkhd\" to finish creating: GET https://rp.j8910848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"name\": \"0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:43.185485913Z\",\n \"endTime\": \"2026-03-13T01:30:19.825677573Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008e25c0>{
msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rnjkhd\" to finish creating: GET https://rp.j8910848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"name\": \"0b86a247-0f2a-4a31-bc1f-16f9237d39d6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:43.185485913Z\",\n \"endTime\": \"2026-03-13T01:30:19.825677573Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-15/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
<*fmt.wrapError | 0xc000874260>:
failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-ql5t9m" to finish creating: GET https://rp.j8736000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa",
"name": "c63d8c71-886e-4886-8a26-caca36c2a4aa",
"status": "Failed",
"startTime": "2026-03-13T01:28:45.12883618Z",
"endTime": "2026-03-13T01:29:53.774390534Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ql5t9m\" to finish creating: GET https://rp.j8736000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"name\": \"c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:45.12883618Z\",\n \"endTime\": \"2026-03-13T01:29:53.774390534Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000874240>{
msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ql5t9m\" to finish creating: GET https://rp.j8736000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"name\": \"c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:45.12883618Z\",\n \"endTime\": \"2026-03-13T01:29:53.774390534Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
<*fmt.wrapError | 0xc000874260>:
failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-ql5t9m" to finish creating: GET https://rp.j8736000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa",
"name": "c63d8c71-886e-4886-8a26-caca36c2a4aa",
"status": "Failed",
"startTime": "2026-03-13T01:28:45.12883618Z",
"endTime": "2026-03-13T01:29:53.774390534Z",
"error": {
"code": "InternalServerError",
"message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ql5t9m\" to finish creating: GET https://rp.j8736000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"name\": \"c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:45.12883618Z\",\n \"endTime\": \"2026-03-13T01:29:53.774390534Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000874240>{
msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ql5t9m\" to finish creating: GET https://rp.j8736000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"name\": \"c63d8c71-886e-4886-8a26-caca36c2a4aa\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-13T01:28:45.12883618Z\",\n \"endTime\": \"2026-03-13T01:29:53.774390534Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------------...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (9)
Affected runs (6) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 5 | 1.32%5 of 380 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 4 · Mar 11: 3 · Mar 12: 1 · Mar 13: 7 · Mar 14: 0 · Mar 15: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc00173f4d0>:
failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-wj68jn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: context deadline exceeded
{
msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-wj68jn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e03920>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn",
},
<*fmt.wrapErrors | 0xc00173f4a0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-wj68jn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e03920>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc00173f4d0>:
failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-wj68jn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: context deadline exceeded
{
msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-wj68jn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e03920>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn",
},
<*fmt.wrapErrors | 0xc00173f4a0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-wj68jn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e03920>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-wj68jn",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapErrors | 0xc000a345d0>:
failed to create HCP cluster list-test-cluster-fg7g2l, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: failed waiting for cluster="list-test-cluster-fg7g2l" in resourcegroup="cluster-listing-nc7jbv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: context deadline exceeded
{
msg: "failed to create HCP cluster list-test-cluster-fg7g2l, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: failed waiting for cluster=\"list-test-cluster-fg7g2l\" in resourcegroup=\"cluster-listing-nc7jbv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea76a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv",
},
<*fmt.wrapErrors | 0xc000a345a0>{
msg: "failed waiting for cluster=\"list-test-cluster-fg7g2l\" in resourcegroup=\"cluster-listing-nc7jbv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea76a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapErrors | 0xc000a345d0>:
failed to create HCP cluster list-test-cluster-fg7g2l, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: failed waiting for cluster="list-test-cluster-fg7g2l" in resourcegroup="cluster-listing-nc7jbv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: context deadline exceeded
{
msg: "failed to create HCP cluster list-test-cluster-fg7g2l, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: failed waiting for cluster=\"list-test-cluster-fg7g2l\" in resourcegroup=\"cluster-listing-nc7jbv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea76a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv",
},
<*fmt.wrapErrors | 0xc000a345a0>{
msg: "failed waiting for cluster=\"list-test-cluster-fg7g2l\" in resourcegroup=\"cluster-listing-nc7jbv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ea76a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-fg7g2l in resource group cluster-listing-nc7jbv",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapErrors | 0xc000a3ac30>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-4pgbh8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: context deadline exceeded
{
msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4pgbh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001224c50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8",
},
<*fmt.wrapErrors | 0xc000a3ac00>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4pgbh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001224c50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapErrors | 0xc000a3ac30>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-4pgbh8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: context deadline exceeded
{
msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4pgbh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001224c50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8",
},
<*fmt.wrapErrors | 0xc000a3ac00>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4pgbh8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001224c50>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-4pgbh8",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (8)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Yet, Eventually failed because the matcher was not satisfied:full failure pattern: Yet, Eventually failed because the matcher was not satisfied: Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 8 | 2.11%8 of 380 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 5 · Mar 13: 2 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.278s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
<v1.PodPhase>: Running
to equal
<v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.278s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
<v1.PodPhase>: Running
to equal
<v1.PodPhase>: Succeededfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.276s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
<v1.PodPhase>: Running
to equal
<v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.276s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
<v1.PodPhase>: Running
to equal
<v1.PodPhase>: Succeededfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.250s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
<v1.PodPhase>: Pending
to equal
<v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.250s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
<v1.PodPhase>: Pending
to equal
<v1.PodPhase>: SucceededContributing tests (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
not to be nilfull failure pattern: not to be nil Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 6 | 1.58%6 of 380 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 6 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: Expected
<*string | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: Expected
<*string | 0x0>: nil
not to be nilContributing tests (1)
Affected runs (6) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 5 | 1.32%5 of 380 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 4 · Mar 11: 2 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapError | 0xc00047e220>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-hmt75w" to finish creating: GET https://rp.j2141696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d",
"name": "deec8042-2914-4e4c-a390-a1755b5f9b9d",
"status": "Failed",
"startTime": "2026-03-11T19:52:56.363794552Z",
"endTime": "2026-03-11T19:54:41.481115713Z",
"error": {
"code": "InternalServerError",
"message": "GET https://arohcpprowoidcj2141696.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:67c1957e-301e-0013-7190-b1fbf4000000\nTime:2026-03-11T19:54:20.9610355Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-hmt75w\" to finish creating: GET https://rp.j2141696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"name\": \"deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T19:52:56.363794552Z\",\n \"endTime\": \"2026-03-11T19:54:41.481115713Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj2141696.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:67c1957e-301e-0013-7190-b1fbf4000000\\nTime:2026-03-11T19:54:20.9610355Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00047e200>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-hmt75w\" to finish creating: GET https://rp.j2141696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"name\": \"deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T19:52:56.363794552Z\",\n \"endTime\": \"2026-03-11T19:54:41.481115713Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj2141696.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:67c1957e-301e-0013-7190-b1fbf4000000\\nTime:2026-03-11T19:54:20.9610355Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapError | 0xc00047e220>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-hmt75w" to finish creating: GET https://rp.j2141696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d",
"name": "deec8042-2914-4e4c-a390-a1755b5f9b9d",
"status": "Failed",
"startTime": "2026-03-11T19:52:56.363794552Z",
"endTime": "2026-03-11T19:54:41.481115713Z",
"error": {
"code": "InternalServerError",
"message": "GET https://arohcpprowoidcj2141696.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:67c1957e-301e-0013-7190-b1fbf4000000\nTime:2026-03-11T19:54:20.9610355Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-hmt75w\" to finish creating: GET https://rp.j2141696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"name\": \"deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T19:52:56.363794552Z\",\n \"endTime\": \"2026-03-11T19:54:41.481115713Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj2141696.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:67c1957e-301e-0013-7190-b1fbf4000000\\nTime:2026-03-11T19:54:20.9610355Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00047e200>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-hmt75w\" to finish creating: GET https://rp.j2141696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"name\": \"deec8042-2914-4e4c-a390-a1755b5f9b9d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T19:52:56.363794552Z\",\n \"endTime\": \"2026-03-11T19:54:41.481115713Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj2141696.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:67c1957e-301e-0013-7190-b1fbf4000000\\nTime:2026-03-11T19:54:20.9610355Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error:
<*fmt.wrapError | 0xc000867300>:
failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-tmtp7t" to finish creating: GET https://rp.j6682624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef",
"name": "6491613a-a2b4-411e-b2bc-43f4c885a5ef",
"status": "Failed",
"startTime": "2026-03-11T17:18:14.777252826Z",
"endTime": "2026-03-11T17:19:49.438177405Z",
"error": {
"code": "InternalServerError",
"message": "GET https://arohcpprowoidcj6682624.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:0728a7cc-301e-0063-1d7b-b13d00000000\nTime:2026-03-11T17:19:44.8267301Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tmtp7t\" to finish creating: GET https://rp.j6682624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"name\": \"6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T17:18:14.777252826Z\",\n \"endTime\": \"2026-03-11T17:19:49.438177405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj6682624.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:0728a7cc-301e-0063-1d7b-b13d00000000\\nTime:2026-03-11T17:19:44.8267301Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008672e0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tmtp7t\" to finish creating: GET https://rp.j6682624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"name\": \"6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T17:18:14.777252826Z\",\n \"endTime\": \"2026-03-11T17:19:49.438177405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj6682624.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:0728a7cc-301e-0063-1d7b-b13d00000000\\nTime:2026-03-11T17:19:44.8267301Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exp...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error:
<*fmt.wrapError | 0xc000867300>:
failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-tmtp7t" to finish creating: GET https://rp.j6682624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef",
"name": "6491613a-a2b4-411e-b2bc-43f4c885a5ef",
"status": "Failed",
"startTime": "2026-03-11T17:18:14.777252826Z",
"endTime": "2026-03-11T17:19:49.438177405Z",
"error": {
"code": "InternalServerError",
"message": "GET https://arohcpprowoidcj6682624.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:0728a7cc-301e-0063-1d7b-b13d00000000\nTime:2026-03-11T17:19:44.8267301Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tmtp7t\" to finish creating: GET https://rp.j6682624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"name\": \"6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T17:18:14.777252826Z\",\n \"endTime\": \"2026-03-11T17:19:49.438177405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj6682624.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:0728a7cc-301e-0063-1d7b-b13d00000000\\nTime:2026-03-11T17:19:44.8267301Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008672e0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tmtp7t\" to finish creating: GET https://rp.j6682624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"name\": \"6491613a-a2b4-411e-b2bc-43f4c885a5ef\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T17:18:14.777252826Z\",\n \"endTime\": \"2026-03-11T17:19:49.438177405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj6682624.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:0728a7cc-301e-0063-1d7b-b13d00000000\\nTime:2026-03-11T17:19:44.8267301Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exp...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapError | 0xc000976620>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-77cmhg" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
"name": "9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
"status": "Failed",
"startTime": "2026-03-10T19:27:23.96442781Z",
"endTime": "2026-03-10T19:29:08.437592376Z",
"error": {
"code": "InternalServerError",
"message": "GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\nTime:2026-03-10T19:28:51.1177193Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000976600>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.R...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapError | 0xc000976620>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-77cmhg" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
"name": "9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
"status": "Failed",
"startTime": "2026-03-10T19:27:23.96442781Z",
"endTime": "2026-03-10T19:29:08.437592376Z",
"error": {
"code": "InternalServerError",
"message": "GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\nTime:2026-03-10T19:28:51.1177193Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000976600>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.R...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (6)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 4 | 1.05%4 of 380 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 42 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | INT, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0011a41e0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-h7jb8n" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000bc7c20>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000bc7c00>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000bc7be0>{
msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0650c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0011a41e0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-h7jb8n" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000bc7c20>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000bc7c00>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000bc7be0>{
msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0650c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000da6e88>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dvxxqx" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc001c19000>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc001c18fe0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc001c18fc0>{
msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0650c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000da6e88>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dvxxqx" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc001c19000>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc001c18fe0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc001c18fc0>{
msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0650c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00020b170>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listqn5zgz" in resourcegroup="ea-list-m4v8xd" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000f891a0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000f89180>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000f89160>{
msg: "failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0650c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00020b170>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listqn5zgz" in resourcegroup="ea-list-m4v8xd" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000f891a0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000f89180>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000f89160>{
msg: "failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0650c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (19)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 4 | 1.05%4 of 380 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 5 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapError | 0xc0012ab180>:
failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while resource is \"provisioning\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000fafb60>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Tue, 10 Mar 2026 19:34:29 GMT",
],
"Content-Length": ["316"],
"X-Envoy-Upstream-Service-Time": ["25"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"f582fe12-33f9-4c9a-95d4-c934cb1441c8",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0012ab0a0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 118, 114, 106, 99, 122, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 316,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3677056.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredenti...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapError | 0xc0012ab180>:
failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while resource is \"provisioning\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000fafb60>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Tue, 10 Mar 2026 19:34:29 GMT",
],
"Content-Length": ["316"],
"X-Envoy-Upstream-Service-Time": ["25"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"f582fe12-33f9-4c9a-95d4-c934cb1441c8",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0012ab0a0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 118, 114, 106, 99, 122, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 316,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3677056.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredenti...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
<*fmt.wrapError | 0xc000a02c20>:
failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while resource is \"provisioning\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f47e60>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"82c53113-ce1b-4dd4-bd27-dd5b5ee92864",
],
"Date": [
"Tue, 10 Mar 2026 19:34:24 GMT",
],
"Content-Length": ["346"],
"X-Envoy-Upstream-Service-Time": ["27"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a02a80>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 122, 98, 116, 50, 109, 108, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 346,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3677056.hcpsvc.osadev.cloud",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
<*fmt.wrapError | 0xc000a02c20>:
failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while resource is \"provisioning\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f47e60>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"82c53113-ce1b-4dd4-bd27-dd5b5ee92864",
],
"Date": [
"Tue, 10 Mar 2026 19:34:24 GMT",
],
"Content-Length": ["346"],
"X-Envoy-Upstream-Service-Time": ["27"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a02a80>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 122, 98, 116, 50, 109, 108, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 346,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3677056.hcpsvc.osadev.cloud",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:173]: Unexpected error:
<*fmt.wrapError | 0xc00141e7a0>:
failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while resource is \"provisioning\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008b7830>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"032dd7ef-e3c7-4688-a613-f8a1b5d25b8e",
],
"Date": [
"Tue, 10 Mar 2026 15:57:48 GMT",
],
"Content-Length": ["346"],
"X-Envoy-Upstream-Service-Time": ["1028"],
},
Body: <*exported.nopClosingBytesReader | 0xc00141e6a0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 101, 115, 116, 45, 104, 120, 106, 107, 56, 109, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 57, 53, 54, 104, 102, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 346,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j1986688.hcpsvc.osadev.cloud",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:173]: Unexpected error:
<*fmt.wrapError | 0xc00141e7a0>:
failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot request credential while resource is \"provisioning\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008b7830>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"032dd7ef-e3c7-4688-a613-f8a1b5d25b8e",
],
"Date": [
"Tue, 10 Mar 2026 15:57:48 GMT",
],
"Content-Length": ["346"],
"X-Envoy-Upstream-Service-Time": ["1028"],
},
Body: <*exported.nopClosingBytesReader | 0xc00141e6a0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 101, 115, 116, 45, 104, 120, 106, 107, 56, 109, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 57, 53, 54, 104, 102, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 346,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j1986688.hcpsvc.osadev.cloud",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (5)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 4 | 1.05%4 of 380 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 5 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapError | 0xc000a0b220>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4xs98c": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a0b200>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000990c90>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"d89e5839-2ec1-4aaa-a7fe-70f6904cc07d",
],
"Date": [
"Tue, 10 Mar 2026 19:34:28 GMT",
],
"Content-Length": ["340"],
"X-Envoy-Upstream-Service-Time": ["2101"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a0b120>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapError | 0xc000a0b220>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4xs98c": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a0b200>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000990c90>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"d89e5839-2ec1-4aaa-a7fe-70f6904cc07d",
],
"Date": [
"Tue, 10 Mar 2026 19:34:28 GMT",
],
"Content-Length": ["340"],
"X-Envoy-Upstream-Service-Time": ["2101"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a0b120>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 1...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
<*fmt.wrapError | 0xc000775300>:
failed to create NodePool nodepool-128GiB: failed starting nodepool creation "nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-g7rf7z": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool nodepool-128GiB: failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007752e0>{
msg: "failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011b5da0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Tue, 10 Mar 2026 15:57:57 GMT",
],
"Content-Length": ["344"],
"X-Envoy-Upstream-Service-Time": ["85"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"7e936490-be89-47eb-a302-7b10d5507757",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000775200>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
<*fmt.wrapError | 0xc000775300>:
failed to create NodePool nodepool-128GiB: failed starting nodepool creation "nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-g7rf7z": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool nodepool-128GiB: failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007752e0>{
msg: "failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011b5da0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Tue, 10 Mar 2026 15:57:57 GMT",
],
"Content-Length": ["344"],
"X-Envoy-Upstream-Service-Time": ["85"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"7e936490-be89-47eb-a302-7b10d5507757",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000775200>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
<*fmt.wrapError | 0xc000a3afe0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-g779ch": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a3afc0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012bad50>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"3707dcaf-af3a-4d6e-bc43-be2c2e5217f8",
],
"Date": [
"Tue, 10 Mar 2026 15:57:56 GMT",
],
"Content-Length": ["345"],
"X-Envoy-Upstream-Service-Time": ["103"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a3aee0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
<*fmt.wrapError | 0xc000a3afe0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-g779ch": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a3afc0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012bad50>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"3707dcaf-af3a-4d6e-bc43-be2c2e5217f8",
],
"Date": [
"Tue, 10 Mar 2026 15:57:56 GMT",
],
"Content-Length": ["345"],
"X-Envoy-Upstream-Service-Time": ["103"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a3aee0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 4 | 1.05%4 of 380 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsMar 8: 0 · Mar 9: 3 · Mar 10: 1 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error:
<*errors.joinError | 0xc001362360>:
cluster listing in non-existent resource group: expected error but none occurred
node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*errors.errorString | 0xc001142210>{
s: "cluster listing in non-existent resource group: expected error but none occurred",
},
<*errors.errorString | 0xc0011431a0>{
s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
<*errors.errorString | 0xc000f366a0>{
s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error:
<*errors.joinError | 0xc001362360>:
cluster listing in non-existent resource group: expected error but none occurred
node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*errors.errorString | 0xc001142210>{
s: "cluster listing in non-existent resource group: expected error but none occurred",
},
<*errors.errorString | 0xc0011431a0>{
s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
<*errors.errorString | 0xc000f366a0>{
s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
<*fmt.wrapError | 0xc000945460>:
failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-w7l2w2" for cluster="ea-cluster": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-w7l2w2\" for cluster=\"ea-cluster\": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0007eb350>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"c61ab021-ab32-4e23-a190-d1d48acb7623",
],
"Date": [
"Mon, 09 Mar 2026 23:29:48 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["112"],
},
Body: <*exported.nopClosingBytesReader | 0xc000944fe0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j9590400.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-09T23:29:39Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
<*fmt.wrapError | 0xc000945460>:
failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-w7l2w2" for cluster="ea-cluster": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-w7l2w2\" for cluster=\"ea-cluster\": PUT https://rp.j9590400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0007eb350>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"c61ab021-ab32-4e23-a190-d1d48acb7623",
],
"Date": [
"Mon, 09 Mar 2026 23:29:48 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["112"],
},
Body: <*exported.nopClosingBytesReader | 0xc000944fe0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j9590400.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w7l2w2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-09T23:29:39Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error:
<*errors.joinError | 0xc000f746f0>:
cluster listing in non-existent resource group: expected error but none occurred
node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*errors.errorString | 0xc0014f2440>{
s: "cluster listing in non-existent resource group: expected error but none occurred",
},
<*errors.errorString | 0xc0014f2c00>{
s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
<*errors.errorString | 0xc0012b70a0>{
s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error:
<*errors.joinError | 0xc000f746f0>:
cluster listing in non-existent resource group: expected error but none occurred
node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*errors.errorString | 0xc0014f2440>{
s: "cluster listing in non-existent resource group: expected error but none occurred",
},
<*errors.errorString | 0xc0014f2c00>{
s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r8rsmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
<*errors.errorString | 0xc0012b70a0>{
s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j6919040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
],
}
occurredContributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Errorfull failure pattern: port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 4 | 1.05%4 of 380 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 3 · Mar 11: 0 · Mar 12: 0 · Mar 13: 1 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc0006c53e0>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc00090cc20>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc0006c53e0>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc00090cc20>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc0000df9e0>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc000746350>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc0000df9e0>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc000746350>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc000894420>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc00069a270>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
<*fmt.wrapError | 0xc000894420>:
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error
{
msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
err: <*errors.errorString | 0xc00069a270>{
s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
},
}
occurredContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
E2E Jobs
46
Success Rate
32.61%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.34.09% (15/44)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 75.56% | 45 | integration/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 77.78% | 45 | integration/parallel | Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle | |
| 86.67% | 15 | integration/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade using the candidate channel | |
| 91.11% | 45 | integration/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 91.11% | 45 | integration/parallel | Customer should create an HCP cluster and validate TLS certificates |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 4 | 8.70%4 of 46 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 0 · Mar 11: 1 · Mar 12: 1 · Mar 13: 0 · Mar 14: 1 · Mar 15: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapErrors | 0xc0009aed80>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-ccshfw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: context deadline exceeded
{
msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ccshfw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f45af0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw",
},
<*fmt.wrapErrors | 0xc0009aed50>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ccshfw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f45af0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapErrors | 0xc0009aed80>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-ccshfw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: context deadline exceeded
{
msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ccshfw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f45af0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw",
},
<*fmt.wrapErrors | 0xc0009aed50>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ccshfw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f45af0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-ccshfw",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapErrors | 0xc00032e8d0>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zkjhtw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded
{
msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0014f4be0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
},
<*fmt.wrapErrors | 0xc00032e8a0>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0014f4be0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapErrors | 0xc00032e8d0>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zkjhtw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded
{
msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0014f4be0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
},
<*fmt.wrapErrors | 0xc00032e8a0>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0014f4be0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapErrors | 0xc0010ea060>:
failed to create HCP cluster list-test-cluster-ccxwqz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: failed waiting for cluster="list-test-cluster-ccxwqz" in resourcegroup="cluster-listing-d94s94" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: context deadline exceeded
{
msg: "failed to create HCP cluster list-test-cluster-ccxwqz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: failed waiting for cluster=\"list-test-cluster-ccxwqz\" in resourcegroup=\"cluster-listing-d94s94\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a83a70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94",
},
<*fmt.wrapErrors | 0xc0010ea030>{
msg: "failed waiting for cluster=\"list-test-cluster-ccxwqz\" in resourcegroup=\"cluster-listing-d94s94\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a83a70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapErrors | 0xc0010ea060>:
failed to create HCP cluster list-test-cluster-ccxwqz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: failed waiting for cluster="list-test-cluster-ccxwqz" in resourcegroup="cluster-listing-d94s94" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: context deadline exceeded
{
msg: "failed to create HCP cluster list-test-cluster-ccxwqz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: failed waiting for cluster=\"list-test-cluster-ccxwqz\" in resourcegroup=\"cluster-listing-d94s94\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a83a70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94",
},
<*fmt.wrapErrors | 0xc0010ea030>{
msg: "failed waiting for cluster=\"list-test-cluster-ccxwqz\" in resourcegroup=\"cluster-listing-d94s94\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a83a70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-ccxwqz in resource group cluster-listing-d94s94",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (3)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 5 | 10.87%5 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 2 · Mar 13: 3 · Mar 14: 0 · Mar 15: 0 | DEV, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001620390>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-vl49pp" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00080bde0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vl49pp\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00080bdc0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vl49pp\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00080bda0>{
msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vl49pp\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001620390>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-vl49pp" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00080bde0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vl49pp\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00080bdc0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vl49pp\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00080bda0>{
msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vl49pp\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000d462a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-kvlw4b" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0008ff140>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kvlw4b\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0008ff120>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kvlw4b\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0008ff100>{
msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kvlw4b\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000d462a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-kvlw4b" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0008ff140>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kvlw4b\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0008ff120>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kvlw4b\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0008ff100>{
msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kvlw4b\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000a38c90>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-2fj9s5" in resourcegroup="cluster-listing-hdwkfd" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0007639e0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-2fj9s5\" in resourcegroup=\"cluster-listing-hdwkfd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0007636c0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-2fj9s5\" in resourcegroup=\"cluster-listing-hdwkfd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000763560>{
msg: "failed waiting for hcpCluster=\"list-test-cluster-2fj9s5\" in resourcegroup=\"cluster-listing-hdwkfd\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000a38c90>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-2fj9s5" in resourcegroup="cluster-listing-hdwkfd" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc0007639e0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-2fj9s5\" in resourcegroup=\"cluster-listing-hdwkfd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0007636c0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-2fj9s5\" in resourcegroup=\"cluster-listing-hdwkfd\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000763560>{
msg: "failed waiting for hcpCluster=\"list-test-cluster-2fj9s5\" in resourcegroup=\"cluster-listing-hdwkfd\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (6)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 5 | 10.87%5 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 2 · Mar 15: 3 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc0010d0990>:
failed waiting for hcpCluster="admin-cred-lifecycle-pq72j7" in resourcegroup="admin-credential-lifecycle-test-h2qqpf" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-pq72j7 in resource group admin-credential-lifecycle-test-h2qqpf, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-pq72j7\" in resourcegroup=\"admin-credential-lifecycle-test-h2qqpf\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-pq72j7 in resource group admin-credential-lifecycle-test-h2qqpf, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ba21f0>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-pq72j7 in resource group admin-credential-lifecycle-test-h2qqpf",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc0010d0990>:
failed waiting for hcpCluster="admin-cred-lifecycle-pq72j7" in resourcegroup="admin-credential-lifecycle-test-h2qqpf" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-pq72j7 in resource group admin-credential-lifecycle-test-h2qqpf, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-pq72j7\" in resourcegroup=\"admin-credential-lifecycle-test-h2qqpf\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-pq72j7 in resource group admin-credential-lifecycle-test-h2qqpf, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ba21f0>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-pq72j7 in resource group admin-credential-lifecycle-test-h2qqpf",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc0015288d0>:
failed waiting for hcpCluster="admin-cred-lifecycle-xxmzq6" in resourcegroup="admin-credential-lifecycle-test-gb2rdr" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-xxmzq6 in resource group admin-credential-lifecycle-test-gb2rdr, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-xxmzq6\" in resourcegroup=\"admin-credential-lifecycle-test-gb2rdr\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-xxmzq6 in resource group admin-credential-lifecycle-test-gb2rdr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000037e40>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-xxmzq6 in resource group admin-credential-lifecycle-test-gb2rdr",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc0015288d0>:
failed waiting for hcpCluster="admin-cred-lifecycle-xxmzq6" in resourcegroup="admin-credential-lifecycle-test-gb2rdr" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-xxmzq6 in resource group admin-credential-lifecycle-test-gb2rdr, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-xxmzq6\" in resourcegroup=\"admin-credential-lifecycle-test-gb2rdr\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-xxmzq6 in resource group admin-credential-lifecycle-test-gb2rdr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000037e40>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-xxmzq6 in resource group admin-credential-lifecycle-test-gb2rdr",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc001148150>:
failed waiting for hcpCluster="admin-cred-lifecycle-7z8gkd" in resourcegroup="admin-credential-lifecycle-test-mf5bt8" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-7z8gkd in resource group admin-credential-lifecycle-test-mf5bt8, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-7z8gkd\" in resourcegroup=\"admin-credential-lifecycle-test-mf5bt8\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-7z8gkd in resource group admin-credential-lifecycle-test-mf5bt8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00050faa0>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-7z8gkd in resource group admin-credential-lifecycle-test-mf5bt8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:193]: Unexpected error:
<*fmt.wrapErrors | 0xc001148150>:
failed waiting for hcpCluster="admin-cred-lifecycle-7z8gkd" in resourcegroup="admin-credential-lifecycle-test-mf5bt8" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-7z8gkd in resource group admin-credential-lifecycle-test-mf5bt8, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-7z8gkd\" in resourcegroup=\"admin-credential-lifecycle-test-mf5bt8\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-7z8gkd in resource group admin-credential-lifecycle-test-mf5bt8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00050faa0>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-7z8gkd in resource group admin-credential-lifecycle-test-mf5bt8",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidIdentityValues; provider Microsoft.ManagedIdentityfull failure pattern: ERROR CODE: InvalidIdentityValues; provider Microsoft.ManagedIdentity Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 4 | 8.70%4 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 4 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
<*exported.ResponseError | 0xc000f97890>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-zpzk8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidIdentityValues
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidIdentityValues",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T151231Z:9c11778c-ff60-487b-a423-651593da8efe",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 4505E2BF16A0421FA071A149B08EF585 Ref B: CHI261140504060 Ref C: 2026-03-11T15:12:31Z",
],
"Cache-Control": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Failure-Cause": ["gateway"],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 11 Mar 2026 15:12:31 GMT",
],
"Pragma": ["no-cache"],
"Content-Length": ["1927"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"9c11778c-ff60-487b-a423-651593da8efe",
],
"X-Ms-Correlation-Request-Id": [
"9c11778c-ff60-487b-a423-651593da8efe",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001100fc0>{
s: "{\"error\":{\"code\":\"InvalidIdentityValues\",\"message\":\"Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests.\"}}",
i: 0,
},
ContentLength: 1927,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/64f0619f-ebc2-4156-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
<*exported.ResponseError | 0xc000f97890>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-zpzk8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidIdentityValues
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidIdentityValues",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T151231Z:9c11778c-ff60-487b-a423-651593da8efe",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 4505E2BF16A0421FA071A149B08EF585 Ref B: CHI261140504060 Ref C: 2026-03-11T15:12:31Z",
],
"Cache-Control": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Failure-Cause": ["gateway"],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 11 Mar 2026 15:12:31 GMT",
],
"Pragma": ["no-cache"],
"Content-Length": ["1927"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"9c11778c-ff60-487b-a423-651593da8efe",
],
"X-Ms-Correlation-Request-Id": [
"9c11778c-ff60-487b-a423-651593da8efe",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001100fc0>{
s: "{\"error\":{\"code\":\"InvalidIdentityValues\",\"message\":\"Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests.\"}}",
i: 0,
},
ContentLength: 1927,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/64f0619f-ebc2-4156-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
<*exported.ResponseError | 0xc0009bd380>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-nld688/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidIdentityValues
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidIdentityValues",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T130855Z:1f5ad0e5-72af-4f34-87a3-cbd696db9711",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Length": ["1937"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Request-Id": [
"1f5ad0e5-72af-4f34-87a3-cbd696db9711",
],
"X-Ms-Correlation-Request-Id": [
"1f5ad0e5-72af-4f34-87a3-cbd696db9711",
],
"X-Msedge-Ref": [
"Ref A: 79888FDC7E984BA8A98CF39D24F6722A Ref B: CH1AA2020610011 Ref C: 2026-03-11T13:08:55Z",
],
"Date": [
"Wed, 11 Mar 2026 13:08:54 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000d91100>{
s: "{\"error\":{\"code\":\"InvalidIdentityValues\",\"message\":\"Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests.\"}}",
i: 0,
},
ContentLength: 1937,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/64f0619f-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
<*exported.ResponseError | 0xc0009bd380>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-nld688/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidIdentityValues
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidIdentityValues",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"Expires": ["-1"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T130855Z:1f5ad0e5-72af-4f34-87a3-cbd696db9711",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Length": ["1937"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Request-Id": [
"1f5ad0e5-72af-4f34-87a3-cbd696db9711",
],
"X-Ms-Correlation-Request-Id": [
"1f5ad0e5-72af-4f34-87a3-cbd696db9711",
],
"X-Msedge-Ref": [
"Ref A: 79888FDC7E984BA8A98CF39D24F6722A Ref B: CH1AA2020610011 Ref C: 2026-03-11T13:08:55Z",
],
"Date": [
"Wed, 11 Mar 2026 13:08:54 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000d91100>{
s: "{\"error\":{\"code\":\"InvalidIdentityValues\",\"message\":\"Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests.\"}}",
i: 0,
},
ContentLength: 1937,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/64f0619f-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
<*exported.ResponseError | 0xc00087cb10>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-q89nkn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidIdentityValues
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidIdentityValues",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Request-Id": [
"5d0f1a83-e9e3-4322-9402-079527a36e43",
],
"X-Ms-Correlation-Request-Id": [
"5d0f1a83-e9e3-4322-9402-079527a36e43",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 7799C63EF4DB4BDE88CFAD7F106271DB Ref B: CHI261140512040 Ref C: 2026-03-11T06:03:42Z",
],
"Pragma": ["no-cache"],
"Content-Length": ["1947"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T060342Z:5d0f1a83-e9e3-4322-9402-079527a36e43",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 11 Mar 2026 06:03:41 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00011d160>{
s: "{\"error\":{\"code\":\"InvalidIdentityValues\",\"message\":\"Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests.\"}}",
i: 0,
},
ContentLength: 1947,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
<*exported.ResponseError | 0xc00087cb10>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-q89nkn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidIdentityValues
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests."
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidIdentityValues",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Request-Id": [
"5d0f1a83-e9e3-4322-9402-079527a36e43",
],
"X-Ms-Correlation-Request-Id": [
"5d0f1a83-e9e3-4322-9402-079527a36e43",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 7799C63EF4DB4BDE88CFAD7F106271DB Ref B: CHI261140512040 Ref C: 2026-03-11T06:03:42Z",
],
"Pragma": ["no-cache"],
"Content-Length": ["1947"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T060342Z:5d0f1a83-e9e3-4322-9402-079527a36e43",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 11 Mar 2026 06:03:41 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00011d160>{
s: "{\"error\":{\"code\":\"InvalidIdentityValues\",\"message\":\"Invalid value for the identities '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-143/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service'. The 'UserAssignedIdentities' property keys should only be empty null during PATCH requests.\"}}",
i: 0,
},
ContentLength: 1947,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
not to be nilfull failure pattern: not to be nil Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 3 | 6.52%3 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 5 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:91]: Expected
<*string | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:91]: Expected
<*string | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: Expected
<*string | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: Expected
<*string | 0x0>: nil
not to be nilContributing tests (2)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 6.52%3 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 1 · Mar 11: 0 · Mar 12: 3 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
<*fmt.wrapError | 0xc00038c420>:
failed creating external auth "ea-list" in resourcegroup="ea-list-9lhb8x" for cluster="ea-listgv59sm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-9lhb8x\" for cluster=\"ea-listgv59sm\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00076e300>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Retry-After": ["10"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/4616b9c4-eae5-42c0-b42e-e60136e4b27a",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 22:53:06 GMT",
],
"Content-Length": ["186"],
"X-Ms-Request-Id": [
"0daaf6b9-dea5-419a-a781-f5d3f0c88c48",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Error-Code": ["LockContention"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: DD91E82BD13A4C8F974F7F6335BA4D4F Ref B: CHI261140506025 Ref C: 2026-03-12T22:52:57Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["10007"],
"X-Ms-Correlation-Request-Id": [
"132f0482-3ab7-4255-98f8-45a73cf0c24c",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T225307Z:132f0482-3ab7-4255-98f8-45a73cf0c24c",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00038c200>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 34, 10, 32, 32, 32, 32, 125, 10, 125]...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
<*fmt.wrapError | 0xc00038c420>:
failed creating external auth "ea-list" in resourcegroup="ea-list-9lhb8x" for cluster="ea-listgv59sm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-9lhb8x\" for cluster=\"ea-listgv59sm\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00076e300>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Retry-After": ["10"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/4616b9c4-eae5-42c0-b42e-e60136e4b27a",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 22:53:06 GMT",
],
"Content-Length": ["186"],
"X-Ms-Request-Id": [
"0daaf6b9-dea5-419a-a781-f5d3f0c88c48",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Error-Code": ["LockContention"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: DD91E82BD13A4C8F974F7F6335BA4D4F Ref B: CHI261140506025 Ref C: 2026-03-12T22:52:57Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["10007"],
"X-Ms-Correlation-Request-Id": [
"132f0482-3ab7-4255-98f8-45a73cf0c24c",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T225307Z:132f0482-3ab7-4255-98f8-45a73cf0c24c",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00038c200>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 34, 10, 32, 32, 32, 32, 125, 10, 125]...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapError | 0xc00148e400>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-h2fml7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00148e3e0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000b78690>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["186"],
"X-Ms-Error-Code": ["LockContention"],
"X-Envoy-Upstream-Service-Time": ["10025"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/ab289601-a1be-4eea-baec-7844c0a021f7",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 22:53:06 GMT",
],
"Content-Type": ["application/json"],
"Retry-After": ["10"],
"X-Ms-Failure-Cause": ["service"],
"X-Msedge-Ref": [
"Ref A: 6E495E82722B47FAB217CA309B5EFC10 Ref B: CHI261140501060 Ref C: 2026-03-12T22:52:57Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"c86fd44b-9640-4399-b837-438b4fc03302",
],
"X-Ms-Correlation-Request-Id": [
"8be5b8ee-e739-4b49-a85a-b05296aaa904",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
<*fmt.wrapError | 0xc00148e400>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-h2fml7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00148e3e0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000b78690>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["186"],
"X-Ms-Error-Code": ["LockContention"],
"X-Envoy-Upstream-Service-Time": ["10025"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/ab289601-a1be-4eea-baec-7844c0a021f7",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 22:53:06 GMT",
],
"Content-Type": ["application/json"],
"Retry-After": ["10"],
"X-Ms-Failure-Cause": ["service"],
"X-Msedge-Ref": [
"Ref A: 6E495E82722B47FAB217CA309B5EFC10 Ref B: CHI261140501060 Ref C: 2026-03-12T22:52:57Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"c86fd44b-9640-4399-b837-438b4fc03302",
],
"X-Ms-Correlation-Request-Id": [
"8be5b8ee-e739-4b49-a85a-b05296aaa904",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0015420c0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc000840400>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008403e0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0013a6450>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Length": ["186"],
"X-Ms-Error-Code": ["LockContention"],
"X-Ms-Request-Id": [
"57007d8e-6281-4f79-87e3-6b62771677b7",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Thu, 12 Mar 2026 20:29:07 GMT",
],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["10009"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9a19b057-0917-4393-a2cf-ff9874abdb4b",
],
"X-Ms-Correlation-Request-Id": [
"cd3510ea-1cec-45db-8f3f-4cac3c6e7c36",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: DC15C07BB4844226852ED55513D0D6F1 Ref B: CHI261140507054 Re...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0015420c0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc000840400>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008403e0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0013a6450>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Length": ["186"],
"X-Ms-Error-Code": ["LockContention"],
"X-Ms-Request-Id": [
"57007d8e-6281-4f79-87e3-6b62771677b7",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Thu, 12 Mar 2026 20:29:07 GMT",
],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["10009"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9a19b057-0917-4393-a2cf-ff9874abdb4b",
],
"X-Ms-Correlation-Request-Id": [
"cd3510ea-1cec-45db-8f3f-4cac3c6e7c36",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: DC15C07BB4844226852ED55513D0D6F1 Ref B: CHI261140507054 Re...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (4)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 0 · Mar 11: 0 · Mar 12: 2 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc00154e1e0>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-hb8wm7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-hb8wm7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001533930>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc00154e1e0>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-hb8wm7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-hb8wm7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001533930>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
<*fmt.wrapErrors | 0xc001502510>:
failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-7rvs2t" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7rvs2t\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000879850>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
<*fmt.wrapErrors | 0xc001502510>:
failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-7rvs2t" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7rvs2t\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000879850>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:173]: Unexpected error:
<*fmt.wrapErrors | 0xc00133e300>:
failed waiting for hcpCluster="admin-cred-lifecycle-htfxlh" in resourcegroup="admin-credential-lifecycle-test-qq776z" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-htfxlh in resource group admin-credential-lifecycle-test-qq776z, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-htfxlh\" in resourcegroup=\"admin-credential-lifecycle-test-qq776z\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-htfxlh in resource group admin-credential-lifecycle-test-qq776z, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00135d130>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-htfxlh in resource group admin-credential-lifecycle-test-qq776z",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:173]: Unexpected error:
<*fmt.wrapErrors | 0xc00133e300>:
failed waiting for hcpCluster="admin-cred-lifecycle-htfxlh" in resourcegroup="admin-credential-lifecycle-test-qq776z" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-htfxlh in resource group admin-credential-lifecycle-test-qq776z, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-htfxlh\" in resourcegroup=\"admin-credential-lifecycle-test-qq776z\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-htfxlh in resource group admin-credential-lifecycle-test-qq776z, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00135d130>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-htfxlh in resource group admin-credential-lifecycle-test-qq776z",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the ...full failure pattern: ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 2 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
<*fmt.wrapError | 0xc000848600>:
failed creating deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-zhtprd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-zhtprd/providers/Microsoft.Resources/deployments/aro-hcp-demo
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplate
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidTemplate",
"message": "Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.",
"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 0,
"linePosition": 0,
"path": ""
}
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-zhtprd\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-zhtprd/providers/Microsoft.Resources/deployments/aro-hcp-demo\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidTemplate\",\n \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n \"additionalInfo\": [\n {\n \"type\": \"TemplateViolation\",\n \"info\": {\n \"lineNumber\": 0,\n \"linePosition\": 0,\n \"path\": \"\"\n }\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00135a990>{
ErrorCode: "InvalidTemplate",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Request-Id": [
"f4d69cdb-d05b-44a9-9a92-6eddb4d52e6e",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260310T160951Z:f4d69cdb-d05b-44a9-9a92-6eddb4d52e6e",
],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"f4d69cdb-d05b-44a9-9a92-6eddb4d52e6e",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Tue, 10 Mar 2026 16:09:50 GMT",
],
"Content-Length": ["542"],
"X-Msedge-Ref": [
"Ref A: 3F20D18F061F48B0A5963AE25F0A156A Ref B: BL2AA2011003040 Ref C: 2026-03-10T16:09:50Z",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000848540>{
s: "{\"error\":{\"code\":\"InvalidTemplate\",\"message\":\"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\"additionalInfo\":[{\"type\":\"TemplateViolation\",\"info\":{\"lineNumber\":0,\"linePosition\":0,\"path\":\"\"}}]}}",
i: 0,
},
ContentLength: 542,
TransferEncoding: nil,
Close: false,
Uncompress...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
<*fmt.wrapError | 0xc000848600>:
failed creating deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-zhtprd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-zhtprd/providers/Microsoft.Resources/deployments/aro-hcp-demo
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplate
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidTemplate",
"message": "Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.",
"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 0,
"linePosition": 0,
"path": ""
}
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-zhtprd\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-zhtprd/providers/Microsoft.Resources/deployments/aro-hcp-demo\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidTemplate\",\n \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n \"additionalInfo\": [\n {\n \"type\": \"TemplateViolation\",\n \"info\": {\n \"lineNumber\": 0,\n \"linePosition\": 0,\n \"path\": \"\"\n }\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00135a990>{
ErrorCode: "InvalidTemplate",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Request-Id": [
"f4d69cdb-d05b-44a9-9a92-6eddb4d52e6e",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260310T160951Z:f4d69cdb-d05b-44a9-9a92-6eddb4d52e6e",
],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"f4d69cdb-d05b-44a9-9a92-6eddb4d52e6e",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Tue, 10 Mar 2026 16:09:50 GMT",
],
"Content-Length": ["542"],
"X-Msedge-Ref": [
"Ref A: 3F20D18F061F48B0A5963AE25F0A156A Ref B: BL2AA2011003040 Ref C: 2026-03-10T16:09:50Z",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000848540>{
s: "{\"error\":{\"code\":\"InvalidTemplate\",\"message\":\"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\"additionalInfo\":[{\"type\":\"TemplateViolation\",\"info\":{\"lineNumber\":0,\"linePosition\":0,\"path\":\"\"}}]}}",
i: 0,
},
ContentLength: 542,
TransferEncoding: nil,
Close: false,
Uncompress...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
<*fmt.wrapError | 0xc0011369c0>:
failed creating deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-6t29q4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-6t29q4/providers/Microsoft.Resources/deployments/aro-hcp-demo
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplate
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidTemplate",
"message": "Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.",
"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 0,
"linePosition": 0,
"path": ""
}
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-6t29q4\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-6t29q4/providers/Microsoft.Resources/deployments/aro-hcp-demo\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidTemplate\",\n \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n \"additionalInfo\": [\n {\n \"type\": \"TemplateViolation\",\n \"info\": {\n \"lineNumber\": 0,\n \"linePosition\": 0,\n \"path\": \"\"\n }\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000eb44e0>{
ErrorCode: "InvalidTemplate",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"6f0859fb-712e-493d-b9f3-1feb834d30d5",
],
"X-Msedge-Ref": [
"Ref A: 0385E8E79B674F5DB2E6A6AC40C9C59D Ref B: CHI261140506042 Ref C: 2026-03-10T11:15:34Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["542"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"6f0859fb-712e-493d-b9f3-1feb834d30d5",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Tue, 10 Mar 2026 11:15:34 GMT",
],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260310T111535Z:6f0859fb-712e-493d-b9f3-1feb834d30d5",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001136900>{
s: "{\"error\":{\"code\":\"InvalidTemplate\",\"message\":\"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\"additionalInfo\":[{\"type\":\"TemplateViolation\",\"info\":{\"lineNumber\":0,\"linePosition\":0,\"path\":\"\"}}]}}",
i: 0,
},
ContentLength: 542,
TransferEncoding: nil,
Close: false,
Uncompress...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
<*fmt.wrapError | 0xc0011369c0>:
failed creating deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-6t29q4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-6t29q4/providers/Microsoft.Resources/deployments/aro-hcp-demo
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplate
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidTemplate",
"message": "Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.",
"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 0,
"linePosition": 0,
"path": ""
}
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-6t29q4\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-6t29q4/providers/Microsoft.Resources/deployments/aro-hcp-demo\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidTemplate\",\n \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n \"additionalInfo\": [\n {\n \"type\": \"TemplateViolation\",\n \"info\": {\n \"lineNumber\": 0,\n \"linePosition\": 0,\n \"path\": \"\"\n }\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000eb44e0>{
ErrorCode: "InvalidTemplate",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"6f0859fb-712e-493d-b9f3-1feb834d30d5",
],
"X-Msedge-Ref": [
"Ref A: 0385E8E79B674F5DB2E6A6AC40C9C59D Ref B: CHI261140506042 Ref C: 2026-03-10T11:15:34Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["542"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"6f0859fb-712e-493d-b9f3-1feb834d30d5",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Tue, 10 Mar 2026 11:15:34 GMT",
],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260310T111535Z:6f0859fb-712e-493d-b9f3-1feb834d30d5",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001136900>{
s: "{\"error\":{\"code\":\"InvalidTemplate\",\"message\":\"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'tags'. The parameters defined in the template are: 'clusterName, identities, openshiftControlPlaneVersionId, openshiftNodePoolVersionId, persistTagValue, usePooledIdentities'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\"additionalInfo\":[{\"type\":\"TemplateViolation\",\"info\":{\"lineNumber\":0,\"linePosition\":0,\"path\":\"\"}}]}}",
i: 0,
},
ContentLength: 542,
TransferEncoding: nil,
Close: false,
Uncompress...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4full failure pattern: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 2 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.003s.
Expected success, but got an error:
<*errors.joinError | 0xc001069d10>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc00052cbc0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0008a6470>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.003s.
Expected success, but got an error:
<*errors.joinError | 0xc001069d10>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc00052cbc0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0008a6470>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.007s.
Expected success, but got an error:
<*errors.joinError | 0xc000f034d0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc00051c6c0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc00073e8e0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.007s.
Expected success, but got an error:
<*errors.joinError | 0xc000f034d0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc00051c6c0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc00073e8e0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}Contributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 9 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
<*fmt.wrapError | 0xc00171c6c0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-g975c2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00171c6a0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0009663c0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["170"],
"X-Ms-Correlation-Request-Id": [
"cf3a6b30-b9f9-40f5-860b-db7a4db99729",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T231200Z:cf3a6b30-b9f9-40f5-860b-db7a4db99729",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 1334633E7ABC4C9D961015632B4669DE Ref B: CHI261140502060 Ref C: 2026-03-12T23:11:59Z",
],
"Date": [
"Thu, 12 Mar 2026 23:11:59 GMT",
],
"X-Envoy-Upstream-Service-Time": ["201"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/67d0f8c8-def2-4e27-b339-fb62d5a3d936",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"f425ba51-7086-4447-8016-171717a6c5ab",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
<*fmt.wrapError | 0xc00171c6c0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-g975c2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00171c6a0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0009663c0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["170"],
"X-Ms-Correlation-Request-Id": [
"cf3a6b30-b9f9-40f5-860b-db7a4db99729",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T231200Z:cf3a6b30-b9f9-40f5-860b-db7a4db99729",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 1334633E7ABC4C9D961015632B4669DE Ref B: CHI261140502060 Ref C: 2026-03-12T23:11:59Z",
],
"Date": [
"Thu, 12 Mar 2026 23:11:59 GMT",
],
"X-Envoy-Upstream-Service-Time": ["201"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/67d0f8c8-def2-4e27-b339-fb62d5a3d936",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"f425ba51-7086-4447-8016-171717a6c5ab",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:114]: Unexpected error:
<*fmt.wrapError | 0xc001408120>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8t5md5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001408100>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000c90fc0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["2649"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T231012Z:014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
],
"X-Msedge-Ref": [
"Ref A: 8080CB14631643F1BD10EDD83999FADD Ref B: CHI261140513054 Ref C: 2026-03-12T23:10:10Z",
],
"Pragma": ["no-cache"],
"Content-Length": ["170"],
"Expires": ["-1"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9e990a7a-bc1c-4fa5-84e6-6d350f3c8cfe",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 23:10:12 GMT",
],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
],
"Cache-Control": ["no-cache"],
"Content-Ty...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:114]: Unexpected error:
<*fmt.wrapError | 0xc001408120>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8t5md5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001408100>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000c90fc0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["2649"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T231012Z:014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
],
"X-Msedge-Ref": [
"Ref A: 8080CB14631643F1BD10EDD83999FADD Ref B: CHI261140513054 Ref C: 2026-03-12T23:10:10Z",
],
"Pragma": ["no-cache"],
"Content-Length": ["170"],
"Expires": ["-1"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9e990a7a-bc1c-4fa5-84e6-6d350f3c8cfe",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 23:10:12 GMT",
],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
],
"Cache-Control": ["no-cache"],
"Content-Ty...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc00087e580>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-mq56rt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00087e560>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e0e090>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"95277bb3-59de-4af9-a6be-d794c22ffea6",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: DECDB7E93FD643FFB79BD2D9D72D81DF Ref B: CHI261140510025 Ref C: 2026-03-12T22:52:58Z",
],
"Pragma": ["no-cache"],
"Content-Length": ["170"],
"Expires": ["-1"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T225305Z:9b9678c6-264f-423d-bafc-7733fb8aa855",
],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Envoy-Upstream-Service-Time": ["7627"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/a50444fe-08f0-4f32-b20a-20c19b8c9e0e",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"9b9678c6-264f-423d-bafc-7733fb8aa855",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc00087e580>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-mq56rt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00087e560>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e0e090>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"95277bb3-59de-4af9-a6be-d794c22ffea6",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: DECDB7E93FD643FFB79BD2D9D72D81DF Ref B: CHI261140510025 Ref C: 2026-03-12T22:52:58Z",
],
"Pragma": ["no-cache"],
"Content-Length": ["170"],
"Expires": ["-1"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T225305Z:9b9678c6-264f-423d-bafc-7733fb8aa855",
],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Envoy-Upstream-Service-Time": ["7627"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/a50444fe-08f0-4f32-b20a-20c19b8c9e0e",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"9b9678c6-264f-423d-bafc-7733fb8aa855",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (9)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; context ; provider Microsoft.Networkfull failure pattern: ERROR CODE: InternalServerError; context ; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 2 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapError | 0xc000c720a0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qkwf4p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619",
"name": "0b05f658-bbb6-452d-b845-92443ec2c619",
"status": "Failed",
"startTime": "2026-03-11T03:36:35.705780329Z",
"endTime": "2026-03-11T03:38:07.574131728Z",
"error": {
"code": "InternalServerError",
"message": "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE UNAVAILABLE\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"message\": \"Operation de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2 not found.\",\n \"details\": []\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-qkwf4p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"name\": \"0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:35.705780329Z\",\n \"endTime\": \"2026-03-11T03:38:07.574131728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2 not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000c72080>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-qkwf4p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"name\": \"0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:35.705780329Z\",\n \"endTime\": \"2026-03-11T03:38:07.574131728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2 not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00144fd10>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapError | 0xc000c720a0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qkwf4p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619",
"name": "0b05f658-bbb6-452d-b845-92443ec2c619",
"status": "Failed",
"startTime": "2026-03-11T03:36:35.705780329Z",
"endTime": "2026-03-11T03:38:07.574131728Z",
"error": {
"code": "InternalServerError",
"message": "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE UNAVAILABLE\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"message\": \"Operation de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2 not found.\",\n \"details\": []\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-qkwf4p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"name\": \"0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:35.705780329Z\",\n \"endTime\": \"2026-03-11T03:38:07.574131728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2 not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000c72080>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-qkwf4p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"name\": \"0b05f658-bbb6-452d-b845-92443ec2c619\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:35.705780329Z\",\n \"endTime\": \"2026-03-11T03:38:07.574131728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation de3f3e34-ee55-4c5b-bdc7-7cbf3170faf2 not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00144fd10>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapError | 0xc0000d9940>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6h5tqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb",
"name": "5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb",
"status": "Failed",
"startTime": "2026-03-11T03:36:37.014555279Z",
"endTime": "2026-03-11T03:38:07.990052924Z",
"error": {
"code": "InternalServerError",
"message": "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/bcedbd48-06f7-4d82-a965-ffd4f43ad6ae\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE UNAVAILABLE\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"message\": \"Operation bcedbd48-06f7-4d82-a965-ffd4f43ad6ae not found.\",\n \"details\": []\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6h5tqt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"name\": \"5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:37.014555279Z\",\n \"endTime\": \"2026-03-11T03:38:07.990052924Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/bcedbd48-06f7-4d82-a965-ffd4f43ad6ae\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation bcedbd48-06f7-4d82-a965-ffd4f43ad6ae not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0000d9900>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6h5tqt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"name\": \"5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:37.014555279Z\",\n \"endTime\": \"2026-03-11T03:38:07.990052924Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/bcedbd48-06f7-4d82-a965-ffd4f43ad6ae\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation bcedbd48-06f7-4d82-a965-ffd4f43ad6ae not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f006c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapError | 0xc0000d9940>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6h5tqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb",
"name": "5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb",
"status": "Failed",
"startTime": "2026-03-11T03:36:37.014555279Z",
"endTime": "2026-03-11T03:38:07.990052924Z",
"error": {
"code": "InternalServerError",
"message": "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/bcedbd48-06f7-4d82-a965-ffd4f43ad6ae\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE UNAVAILABLE\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"message\": \"Operation bcedbd48-06f7-4d82-a965-ffd4f43ad6ae not found.\",\n \"details\": []\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6h5tqt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"name\": \"5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:37.014555279Z\",\n \"endTime\": \"2026-03-11T03:38:07.990052924Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/bcedbd48-06f7-4d82-a965-ffd4f43ad6ae\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation bcedbd48-06f7-4d82-a965-ffd4f43ad6ae not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0000d9900>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6h5tqt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"name\": \"5c49f2a2-a1d3-45dc-b6e8-265bbbe3eadb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T03:36:37.014555279Z\",\n \"endTime\": \"2026-03-11T03:38:07.990052924Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Network/locations/uksouth/operations/bcedbd48-06f7-4d82-a965-ffd4f43ad6ae\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE UNAVAILABLE\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"message\\\": \\\"Operation bcedbd48-06f7-4d82-a965-ffd4f43ad6ae not found.\\\",\\n \\\"details\\\": []\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f006c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Interrupted by Userfull failure pattern: Interrupted by User Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [:0]: Interrupted by User fail [:0]: Interrupted by User Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0007b0fa0>:
failed to create NodePool arm64-vm-np-1: failed starting nodepool creation "arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-8h7xd7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool arm64-vm-np-1: failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8h7xd7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007b0f60>{
msg: "failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8h7xd7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015e0990>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: 1A43493D763743078F50D58A7107C67A Ref B: CHI261140511036 Ref C: 2026-03-09T15:39:54Z",
],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["348"],
"X-Ms-Request-Id": [
"b47e62cf-b184-4859-bc06-a546313b427d",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T153955Z:71f1c6be-0050-4562-ae53-b90c10721c0e",
],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Envoy-Upstream-Service-Time": ["169"],
"X-Ms-Correlation-Request-Id": [
"71f1c6be-0050-4562-ae53-b90c10721c0e",
],
"Date": [
"Mon, 09 Mar 2026 15:39:54 GMT",
],
"Expires": ["-1"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=64dc69e4-d083-49fc-9569-ebece...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0007b0fa0>:
failed to create NodePool arm64-vm-np-1: failed starting nodepool creation "arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-8h7xd7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool arm64-vm-np-1: failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8h7xd7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0007b0f60>{
msg: "failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8h7xd7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8h7xd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015e0990>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: 1A43493D763743078F50D58A7107C67A Ref B: CHI261140511036 Ref C: 2026-03-09T15:39:54Z",
],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["348"],
"X-Ms-Request-Id": [
"b47e62cf-b184-4859-bc06-a546313b427d",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T153955Z:71f1c6be-0050-4562-ae53-b90c10721c0e",
],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Envoy-Upstream-Service-Time": ["169"],
"X-Ms-Correlation-Request-Id": [
"71f1c6be-0050-4562-ae53-b90c10721c0e",
],
"Date": [
"Mon, 09 Mar 2026 15:39:54 GMT",
],
"Expires": ["-1"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=64dc69e4-d083-49fc-9569-ebece...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Admin credential 1 should fail after revocation, last error: <nil>full failure pattern: Admin credential 1 should fail after revocation, last error: <nil> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 1 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 1 should fail after revocation, last error: <nil>
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 1 should fail after revocation, last error: <nil>
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Resource is already deleting; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Resource is already deleting; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000d78048>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Resource is already deleting",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc001509a00>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0015099e0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0009cc510>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["295"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["3000"],
"X-Ms-Correlation-Request-Id": [
"36e4392a-0d35-4147-8a9f-6300ad30a25c",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T050748Z:36e4392a-0d35-4147-8a9f-6300ad30a25c",
],
"Cache-Control": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"9240eef9-4396-4070-b424-2e847f902062",
],
"X-Envoy-Upstream-Service-Time": ["32460"],
"X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["200"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Mon, 09 Mar 2026 05:07:47 GMT",
],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/1795cff2-640d-4155-ab23-dd61fa932a34",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000d78048>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Resource is already deleting",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc001509a00>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0015099e0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-sqdgm5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-vzqchl\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0009cc510>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["295"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["3000"],
"X-Ms-Correlation-Request-Id": [
"36e4392a-0d35-4147-8a9f-6300ad30a25c",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T050748Z:36e4392a-0d35-4147-8a9f-6300ad30a25c",
],
"Cache-Control": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"9240eef9-4396-4070-b424-2e847f902062",
],
"X-Envoy-Upstream-Service-Time": ["32460"],
"X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["200"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Mon, 09 Mar 2026 05:07:47 GMT",
],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/1795cff2-640d-4155-ab23-dd61fa932a34",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code InvalidRequestContent; detail message Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'full failure pattern: ERROR CODE: DeploymentFailed; detail code InvalidRequestContent; detail message Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8' Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 1 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
<*fmt.wrapError | 0xc000fc6740>:
failed waiting for deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-w6hmz9" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InvalidRequestContent\",\r\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-w6hmz9\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"InvalidRequestContent\\\",\\r\\n \\\"message\\\": \\\"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001500e10>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Content-Length": ["1243"],
"X-Ms-Request-Id": [
"0d231ad8-8318-4e3a-920c-9541e2affd70",
],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"0d231ad8-8318-4e3a-920c-9541e2affd70",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T225347Z:0d231ad8-8318-4e3a-920c-9541e2affd70",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: 4733BC488DE8424898A45EF16542E708 Ref B: CH1AA2020603045 Ref C: 2026-03-12T22:53:47Z",
],
"Date": [
"Thu, 12 Mar 2026 22:53:46 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000fc6620>{
s:...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
<*fmt.wrapError | 0xc000fc6740>:
failed waiting for deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-w6hmz9" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InvalidRequestContent\",\r\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-w6hmz9\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"InvalidRequestContent\\\",\\r\\n \\\"message\\\": \\\"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001500e10>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Content-Length": ["1243"],
"X-Ms-Request-Id": [
"0d231ad8-8318-4e3a-920c-9541e2affd70",
],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"0d231ad8-8318-4e3a-920c-9541e2affd70",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260312T225347Z:0d231ad8-8318-4e3a-920c-9541e2affd70",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: 4733BC488DE8424898A45EF16542E708 Ref B: CH1AA2020603045 Ref C: 2026-03-12T22:53:47Z",
],
"Date": [
"Thu, 12 Mar 2026 22:53:46 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000fc6620>{
s:...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapError | 0xc0012ecd00>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9wh75v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9wh75v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00130de60>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"aded1fbf-5095-4f14-957e-7e4810dfa7b7",
],
"X-Envoy-Upstream-Service-Time": ["1537"],
"Cache-Control": ["no-cache"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/ffa8e515-a5e8-40aa-9ef9-9df55097710b",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T155315Z:1e0a595f-913b-42ed-b06c-d8a52395722d",
],
"X-Msedge-Ref": [
"Ref A: D4B3120337FD4E1A835EB7502426C822 Ref B: CH1AA2020603037 Ref C: 2026-03-09T15:53:13Z",
],
"Date": [
"Mon, 09 Mar 2026 15:53:14 GMT",
],
"Content-Length": ["107"],
"X-Ms-Correlation-Request-Id": [
"1e0a595f-913b-42ed-b06c-d8a52395722d",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
},
Body: <*exported.nopClosingBytesReader | 0xc0012ecc40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
H...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapError | 0xc0012ecd00>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9wh75v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9wh75v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00130de60>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"aded1fbf-5095-4f14-957e-7e4810dfa7b7",
],
"X-Envoy-Upstream-Service-Time": ["1537"],
"Cache-Control": ["no-cache"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/ffa8e515-a5e8-40aa-9ef9-9df55097710b",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T155315Z:1e0a595f-913b-42ed-b06c-d8a52395722d",
],
"X-Msedge-Ref": [
"Ref A: D4B3120337FD4E1A835EB7502426C822 Ref B: CH1AA2020603037 Ref C: 2026-03-09T15:53:13Z",
],
"Date": [
"Mon, 09 Mar 2026 15:53:14 GMT",
],
"Content-Length": ["107"],
"X-Ms-Correlation-Request-Id": [
"1e0a595f-913b-42ed-b06c-d8a52395722d",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
},
Body: <*exported.nopClosingBytesReader | 0xc0012ecc40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
H...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Should receive valid HTTP status code from authorized IPfull failure pattern: Should receive valid HTTP status code from authorized IP Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 1 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:167 with:
Should receive valid HTTP status code from authorized IP
Expected
<string>: 000
to match regular expression
<string>: ^[2-5][0-9][0-9]$
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:167 with:
Should receive valid HTTP status code from authorized IP
Expected
<string>: 000
to match regular expression
<string>: ^[2-5][0-9][0-9]$Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Timed out after 120.345s.full failure pattern: Timed out after 120.345s. Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 1 · Mar 15: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.345s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:162 with:
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.345s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:162 with:
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
client rate limiter Wait returned an error: context deadline exceededfull failure pattern: client rate limiter Wait returned an error: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 1 should fail after revocation, last error: client rate limiter Wait returned an error: context deadline exceeded
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: Admin credential 1 should fail after revocation, last error: client rate limiter Wait returned an error: context deadline exceeded
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
E2E Jobs
20
Success Rate
60.00%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.66.67% (12/18)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 72.73% | 11 | stage/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade using the candidate channel | |
| 77.78% | 18 | stage/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 83.33% | 18 | stage/parallel | Customer should be able to create an HCP cluster and custom node pool osDisk size | |
| 83.33% | 18 | stage/parallel | Customer should be able to create node pools with ARM64-based VMs | |
| 83.33% | 18 | stage/parallel | Customer should create an HCP cluster and validate TLS certificates |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 4 | 20.00%4 of 20 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 2 · Mar 13: 4 · Mar 14: 0 · Mar 15: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0017ea2e8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-f85kxr" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000d15e00>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-f85kxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d15de0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-f85kxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d15dc0>{
msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-f85kxr\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0017ea2e8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-f85kxr" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000d15e00>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-f85kxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d15de0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-f85kxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d15dc0>{
msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-f85kxr\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000da81c8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-bbt9j2" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000b35120>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bbt9j2\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b35100>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bbt9j2\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b350e0>{
msg: "failed waiting for hcpCluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bbt9j2\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000da81c8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-bbt9j2" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000b35120>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bbt9j2\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b35100>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bbt9j2\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b350e0>{
msg: "failed waiting for hcpCluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bbt9j2\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0015623f0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listcwcr6n" in resourcegroup="ea-list-kd7958" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc001681600>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listcwcr6n\" in resourcegroup=\"ea-list-kd7958\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0016815e0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listcwcr6n\" in resourcegroup=\"ea-list-kd7958\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0016815c0>{
msg: "failed waiting for hcpCluster=\"ea-listcwcr6n\" in resourcegroup=\"ea-list-kd7958\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0015623f0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listcwcr6n" in resourcegroup="ea-list-kd7958" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc001681600>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listcwcr6n\" in resourcegroup=\"ea-list-kd7958\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0016815e0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listcwcr6n\" in resourcegroup=\"ea-list-kd7958\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc0016815c0>{
msg: "failed waiting for hcpCluster=\"ea-listcwcr6n\" in resourcegroup=\"ea-list-kd7958\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb08a0c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (5)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 3 | 15.00%3 of 20 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 4 · Mar 14: 0 · Mar 15: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
<*fmt.wrapError | 0xc000bea540>:
failed to create NodePool arm64-vm-np-1: failed starting nodepool creation "arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-d9rlj5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-d9rlj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool arm64-vm-np-1: failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-d9rlj5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-d9rlj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bea520>{
msg: "failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-d9rlj5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-d9rlj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012ae630>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"Retry-After": ["10"],
"X-Envoy-Upstream-Service-Time": ["10019"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: ADF6D0781C9D48D8A5C81735B6310D6F Ref B: CHI261140512034 Ref C: 2026-03-13T07:15:12Z",
],
"Date": [
"Fri, 13 Mar 2026 07:15:21 GMT",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6b3f3604-a2f9-4117-8f92-506228f3e7af",
],
"Pragma": ["no-cache"],
"Content-Length": ["186"],
"X-Ms-Request-Id": [
"8c39947a-1214-430d-8010-f0929133b44f",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"dafc4310-6617-45a5-b396-fc80a09e00a8",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
<*fmt.wrapError | 0xc000bea540>:
failed to create NodePool arm64-vm-np-1: failed starting nodepool creation "arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-d9rlj5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-d9rlj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool arm64-vm-np-1: failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-d9rlj5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-d9rlj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bea520>{
msg: "failed starting nodepool creation \"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-d9rlj5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-d9rlj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012ae630>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"Retry-After": ["10"],
"X-Envoy-Upstream-Service-Time": ["10019"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: ADF6D0781C9D48D8A5C81735B6310D6F Ref B: CHI261140512034 Ref C: 2026-03-13T07:15:12Z",
],
"Date": [
"Fri, 13 Mar 2026 07:15:21 GMT",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6b3f3604-a2f9-4117-8f92-506228f3e7af",
],
"Pragma": ["no-cache"],
"Content-Length": ["186"],
"X-Ms-Request-Id": [
"8c39947a-1214-430d-8010-f0929133b44f",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"dafc4310-6617-45a5-b396-fc80a09e00a8",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000fd40f0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-26rkfj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc001786300>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-26rkfj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0017862e0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-26rkfj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0013d0540>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: CC4F23FE193F4C1FB16C204ED434C9F4 Ref B: CHI261140510042 Ref C: 2026-03-13T03:16:52Z",
],
"Expires": ["-1"],
"Retry-After": ["10"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/fbb98992-521d-40e7-b146-c0dfb4a73360",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
"Date": [
"Fri, 13 Mar 2026 03:17:01 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["LockContention"],
"X-Ms-Correlation-Request-Id": [
"f2fd7682-45a9-47f9-a802-ee0323a08a2f",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T031702Z:f2fd7682-45a9-47f9-a802-ee0323a08a2f",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000fd40f0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-26rkfj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc001786300>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-26rkfj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0017862e0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-26rkfj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0013d0540>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: CC4F23FE193F4C1FB16C204ED434C9F4 Ref B: CHI261140510042 Ref C: 2026-03-13T03:16:52Z",
],
"Expires": ["-1"],
"Retry-After": ["10"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/fbb98992-521d-40e7-b146-c0dfb4a73360",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
"Date": [
"Fri, 13 Mar 2026 03:17:01 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": ["LockContention"],
"X-Ms-Correlation-Request-Id": [
"f2fd7682-45a9-47f9-a802-ee0323a08a2f",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T031702Z:f2fd7682-45a9-47f9-a802-ee0323a08a2f",
],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000d300d8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-v4pw7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc000e024c0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-v4pw7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000e02480>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-v4pw7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f90300>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Fri, 13 Mar 2026 03:19:28 GMT",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Retry-After": ["10"],
"X-Envoy-Upstream-Service-Time": ["10022"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"8b4fb22e-bc20-46b1-8f98-d4392041161d",
],
"X-Msedge-Ref": [
"Ref A: A9CAA84EC1294F488AB1EBE718FAA146 Ref B: CH1AA2020603009 Ref C: 2026-03-13T03:19:18Z",
],
"Content-Length": ["186"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"51a5e370-92d6-4444-ab52-1b5412da1e41",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/c7e2a0d7-ccf1-4903-af7f-e49949aa8449",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T031929Z:8b4fb22e-bc20-46b1-8f98-d4392041161d",
],
"Strict-Transport-Security": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000d300d8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-v4pw7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: LockContention
--------------------------------------------------------------------------------
{
"error": {
"code": "LockContention",
"message": "Failed to acquire lock: timed out",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc000e024c0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-v4pw7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000e02480>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-v4pw7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f90300>{
ErrorCode: "LockContention",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Fri, 13 Mar 2026 03:19:28 GMT",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Retry-After": ["10"],
"X-Envoy-Upstream-Service-Time": ["10022"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"8b4fb22e-bc20-46b1-8f98-d4392041161d",
],
"X-Msedge-Ref": [
"Ref A: A9CAA84EC1294F488AB1EBE718FAA146 Ref B: CH1AA2020603009 Ref C: 2026-03-13T03:19:18Z",
],
"Content-Length": ["186"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"51a5e370-92d6-4444-ab52-1b5412da1e41",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/c7e2a0d7-ccf1-4903-af7f-e49949aa8449",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T031929Z:8b4fb22e-bc20-46b1-8f98-d4392041161d",
],
"Strict-Transport-Security": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4full failure pattern: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 15.00%3 of 20 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 1 · Mar 13: 2 · Mar 14: 0 · Mar 15: 0 | INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.005s.
Expected success, but got an error:
<*errors.joinError | 0xc001085770>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc000a12720>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0008a2660>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.005s.
Expected success, but got an error:
<*errors.joinError | 0xc001085770>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc000a12720>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0008a2660>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.009s.
Expected success, but got an error:
<*errors.joinError | 0xc0011582e8>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc000ac67c0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0001427d0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.009s.
Expected success, but got an error:
<*errors.joinError | 0xc0011582e8>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc000ac67c0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0001427d0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.008s.
Expected success, but got an error:
<*errors.joinError | 0xc000c0f0f8>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc0002fc120>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc000662dd0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.008s.
Expected success, but got an error:
<*errors.joinError | 0xc000c0f0f8>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc0002fc120>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc000662dd0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 10.00%2 of 20 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 10 · Mar 13: 10 · Mar 14: 0 · Mar 15: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
<*fmt.wrapError | 0xc00061c6e0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-s58clz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-s58clz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-s58clz\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-s58clz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00061c680>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-s58clz\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-s58clz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000c48660>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: 464A75CC61A1478A90E29E6DFB49BE37 Ref B: CH1AA2020610029 Ref C: 2026-03-13T01:49:31Z",
],
"Content-Length": ["170"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"aed4189a-5711-491e-8bb6-acfacec34220",
],
"X-Envoy-Upstream-Service-Time": ["286"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T014932Z:5707dfab-8c93-438a-85f8-fe0b471a077a",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Pragma": ["no-cache"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/307313a4-b300-4bb0-b030-114e8e644174",
],
"Date": [
"Fri, 13 Mar 2026 01:49:31 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"5707dfab-8c93-438a-85f8-fe0b471a077a",
],
"Strict-Transpo...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
<*fmt.wrapError | 0xc00061c6e0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-s58clz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-s58clz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-s58clz\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-s58clz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00061c680>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-s58clz\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-s58clz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000c48660>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: 464A75CC61A1478A90E29E6DFB49BE37 Ref B: CH1AA2020610029 Ref C: 2026-03-13T01:49:31Z",
],
"Content-Length": ["170"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"aed4189a-5711-491e-8bb6-acfacec34220",
],
"X-Envoy-Upstream-Service-Time": ["286"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T014932Z:5707dfab-8c93-438a-85f8-fe0b471a077a",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Pragma": ["no-cache"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/307313a4-b300-4bb0-b030-114e8e644174",
],
"Date": [
"Fri, 13 Mar 2026 01:49:31 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Content-Type-Options": ["nosniff"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"5707dfab-8c93-438a-85f8-fe0b471a077a",
],
"Strict-Transpo...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
<*fmt.wrapError | 0xc000d94240>:
failed to create NodePool nodepool-128GiB: failed starting nodepool creation "nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-8cnnck": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8cnnck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool nodepool-128GiB: failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8cnnck\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8cnnck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000d94220>{
msg: "failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8cnnck\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8cnnck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011dc330>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: CE27CC750B4F4A1099A8FE01F39FD561 Ref B: CHI261140504029 Ref C: 2026-03-13T01:48:41Z",
],
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"3a2f44cb-0316-432e-9f20-f948736553c2",
],
"X-Ms-Correlation-Request-Id": [
"e54bd54b-460d-43e4-a17a-c650ba3ec54f",
],
"X-Envoy-Upstream-Service-Time": ["3358"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/26c75988-1220-4415-af03-21c025e4c013",
],
"Date": [
"Fri, 13 Mar 2026 01:48:44 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["170"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T014844Z:e54bd54b-460d-43e4-a17a-c650ba3ec54f",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"P...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
<*fmt.wrapError | 0xc000d94240>:
failed to create NodePool nodepool-128GiB: failed starting nodepool creation "nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-8cnnck": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8cnnck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool nodepool-128GiB: failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8cnnck\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8cnnck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000d94220>{
msg: "failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8cnnck\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8cnnck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011dc330>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: CE27CC750B4F4A1099A8FE01F39FD561 Ref B: CHI261140504029 Ref C: 2026-03-13T01:48:41Z",
],
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"3a2f44cb-0316-432e-9f20-f948736553c2",
],
"X-Ms-Correlation-Request-Id": [
"e54bd54b-460d-43e4-a17a-c650ba3ec54f",
],
"X-Envoy-Upstream-Service-Time": ["3358"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/26c75988-1220-4415-af03-21c025e4c013",
],
"Date": [
"Fri, 13 Mar 2026 01:48:44 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["170"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T014844Z:e54bd54b-460d-43e4-a17a-c650ba3ec54f",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"P...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc000a03e00>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-fph99j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fph99j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fph99j\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fph99j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a03de0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fph99j\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fph99j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015b46f0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["170"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["2485"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T014759Z:b223bd31-5124-4336-8659-d621923b7581",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"985c3873-8557-481e-b7da-f41abf57b299",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/223eff5f-1b5c-4425-89eb-063d6fc5f956",
],
"X-Ms-Correlation-Request-Id": [
"b223bd31-5124-4336-8659-d621923b7581",
],
"X-Msedge-Ref": [
"Ref A: B9DD73C420794F7A8138A30F859A721A Ref B: CHI261140505042 Ref C: 2026-03-13T01:47:56Z",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc000a03e00>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-fph99j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fph99j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fph99j\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fph99j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000a03de0>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fph99j\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fph99j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015b46f0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["170"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["2485"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260313T014759Z:b223bd31-5124-4336-8659-d621923b7581",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"985c3873-8557-481e-b7da-f41abf57b299",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/223eff5f-1b5c-4425-89eb-063d6fc5f956",
],
"X-Ms-Correlation-Request-Id": [
"b223bd31-5124-4336-8659-d621923b7581",
],
"X-Msedge-Ref": [
"Ref A: B9DD73C420794F7A8138A30F859A721A Ref B: CHI261140505042 Ref C: 2026-03-13T01:47:56Z",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (10)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Unauthorizedfull failure pattern: Unauthorized Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.00%1 of 20 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 1 · Mar 15: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:222]: Timed out after 300.007s.
Expected success, but got an error:
<*errors.StatusError | 0xc001083400>:
Unauthorized
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:222]: Timed out after 300.007s.
Expected success, but got an error:
<*errors.StatusError | 0xc001083400>:
Unauthorized
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=False, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pr...full failure pattern: cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=False, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.00%1 of 20 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 1 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
<*errors.errorString | 0xc000efdf00>:
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=False, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)
{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=False, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
<*errors.errorString | 0xc000efdf00>:
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=False, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)
{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=False, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)",
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
E2E Jobs
40
Success Rate
67.50%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.71.05% (27/38)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 89.47% | 38 | prod/parallel | Customer should be able to list HCP clusters without node pools at both subscription and resource group levels | |
| 90.91% | 11 | prod/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade using the candidate channel | |
| 92.11% | 38 | prod/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 94.74% | 38 | prod/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 94.74% | 38 | prod/parallel | Customer should be able to create an HCP cluster |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Interrupted by Userfull failure pattern: Interrupted by User Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 4 | 10.00%4 of 40 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 1 · Mar 11: 2 · Mar 12: 1 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV, INT | |||||||||||||||||||||||||||||||
Full failure examples (1)fail [:0]: Interrupted by User fail [:0]: Interrupted by User Contributing tests (3)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 3 | 7.50%3 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 1 · Mar 10: 1 · Mar 11: 1 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV, INT | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc0008462a0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-jblfjr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jblfjr\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000846220>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jblfjr\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015d4600>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/d6c3e6e1-a3e3-4ef3-80a7-7d6e66e285d7",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T011957Z:ced5b521-bd7b-4f51-87af-95c4f398bece",
],
"Content-Length": ["342"],
"Expires": ["-1"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"a82fb0c7-45ed-4b00-881a-cdced9a42af5",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["1147"],
"X-Ms-Correlation-Request-Id": [
"ced5b521-bd7b-4f51-87af-95c4f398bece",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc0008462a0>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-jblfjr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jblfjr\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000846220>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jblfjr\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jblfjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015d4600>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/d6c3e6e1-a3e3-4ef3-80a7-7d6e66e285d7",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260311T011957Z:ced5b521-bd7b-4f51-87af-95c4f398bece",
],
"Content-Length": ["342"],
"Expires": ["-1"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Request-Id": [
"a82fb0c7-45ed-4b00-881a-cdced9a42af5",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Type": ["application/json"],
"X-Envoy-Upstream-Service-Time": ["1147"],
"X-Ms-Correlation-Request-Id": [
"ced5b521-bd7b-4f51-87af-95c4f398bece",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
<*fmt.wrapError | 0xc001686ec0>:
failed creating external auth "ea-list" in resourcegroup="ea-list-kthdbg" for cluster="ea-listrzg728": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list"
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-kthdbg\" for cluster=\"ea-listrzg728\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00078ade0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["330"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Correlation-Request-Id": [
"76f305c5-fa69-4ac9-9551-ae1c49be1677",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Envoy-Upstream-Service-Time": ["192"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/927d5101-f4e6-4dbb-b7b6-eedffc1e63fb",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260310T021748Z:76f305c5-fa69-4ac9-9551-ae1c49be1677",
],
"X-Msedge-Ref": [
"Ref A: DE3DCB908678429D885260478E07FD22 Ref B: CHI261140506052 Ref C: 2026-03-10T02:17:47Z",
],
"Pragma": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Date": [
"Tue, 10 Mar 2026 02:17:48 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"87a9e39a-4013-4c2d-9c4c-6c3c6dcebe75",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001686de0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 52, 48, 51, 100, 57, 100, 101, 57, 45, 49, 51, 50, 98, 45, 52, 57, 55, 52, 45, 57...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
<*fmt.wrapError | 0xc001686ec0>:
failed creating external auth "ea-list" in resourcegroup="ea-list-kthdbg" for cluster="ea-listrzg728": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list"
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-kthdbg\" for cluster=\"ea-listrzg728\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-kthdbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listrzg728/externalAuths/ea-list\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00078ade0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["330"],
"X-Ms-Error-Code": ["Conflict"],
"X-Ms-Correlation-Request-Id": [
"76f305c5-fa69-4ac9-9551-ae1c49be1677",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Envoy-Upstream-Service-Time": ["192"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/927d5101-f4e6-4dbb-b7b6-eedffc1e63fb",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260310T021748Z:76f305c5-fa69-4ac9-9551-ae1c49be1677",
],
"X-Msedge-Ref": [
"Ref A: DE3DCB908678429D885260478E07FD22 Ref B: CHI261140506052 Ref C: 2026-03-10T02:17:47Z",
],
"Pragma": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Date": [
"Tue, 10 Mar 2026 02:17:48 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"87a9e39a-4013-4c2d-9c4c-6c3c6dcebe75",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001686de0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 52, 48, 51, 100, 57, 100, 101, 57, 45, 49, 51, 50, 98, 45, 52, 57, 55, 52, 45, 57...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc000796180>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-g9dmqs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-g9dmqs\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000796160>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-g9dmqs\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f002d0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Length": ["342"],
"X-Ms-Request-Id": [
"3390720e-9df1-4ec3-bca3-5b8c08d9aad9",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T023420Z:0ae2cd1e-878a-414b-83bd-de591313a919",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: F525233478294E04B588F5D72E44E37B Ref B: CHI261140506023 Ref C: 2026-03-09T02:34:20Z",
],
"Pragma": ["no-cache"],
"X-Ms-Error-Code": ["Conflict"],
"X-Envoy-Upstream-Service-Time": ["126"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"0ae2cd1e-878a-414b-83bd-de591313a919",
],
"Expires": ["-1"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/ac773c95-5ddf-4064...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
<*fmt.wrapError | 0xc000796180>:
failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-g9dmqs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot create resource while parent resource is provisioning",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-g9dmqs\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000796160>{
msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-g9dmqs\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot create resource while parent resource is provisioning\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-g9dmqs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000f002d0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Content-Length": ["342"],
"X-Ms-Request-Id": [
"3390720e-9df1-4ec3-bca3-5b8c08d9aad9",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260309T023420Z:0ae2cd1e-878a-414b-83bd-de591313a919",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: F525233478294E04B588F5D72E44E37B Ref B: CHI261140506023 Ref C: 2026-03-09T02:34:20Z",
],
"Pragma": ["no-cache"],
"X-Ms-Error-Code": ["Conflict"],
"X-Envoy-Upstream-Service-Time": ["126"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"0ae2cd1e-878a-414b-83bd-de591313a919",
],
"Expires": ["-1"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/ac773c95-5ddf-4064...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||
ERROR CODE: ResourceGroupDeletionFailedfull failure pattern: ERROR CODE: ResourceGroupDeletionFailed Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 7.50%3 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 2 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc001716168>:
failed to cleanup resource group: failed waiting for resourcegroup="external-auth-cluster-b67ng8" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1FWFRFUk5BTDoyREFVVEg6MkRDTFVTVEVSOjJEQjY3Tkc4LUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'external-auth-cluster-b67ng8' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc000f5e3a0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"external-auth-cluster-b67ng8\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1FWFRFUk5BTDoyREFVVEg6MkRDTFVTVEVSOjJEQjY3Tkc4LUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'external-auth-cluster-b67ng8' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f5e380>{
msg: "failed waiting for resourcegroup=\"external-auth-cluster-b67ng8\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1FWFRFUk5BTDoyREFVVEg6MkRDTFVTVEVSOjJEQjY3Tkc4LUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'external-auth-cluster-b67ng8' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001332a50>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Correlation-Request-Id": [
"8a7a4e2c-0c5c-420e-a471-80fa19b53da3",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 04:04:21 GMT",
],
"Content-Length": ["247"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260312T040421Z:8a7a4e2c-0c5c-420e-a471-80fa19b53da3",
],
"X-Ms-Request-Id": [
"8a7a4e2c-0c5c-420e-a471-80fa19b53da3",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: D8B73840D3704...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc001716168>:
failed to cleanup resource group: failed waiting for resourcegroup="external-auth-cluster-b67ng8" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1FWFRFUk5BTDoyREFVVEg6MkRDTFVTVEVSOjJEQjY3Tkc4LUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'external-auth-cluster-b67ng8' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc000f5e3a0>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"external-auth-cluster-b67ng8\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1FWFRFUk5BTDoyREFVVEg6MkRDTFVTVEVSOjJEQjY3Tkc4LUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'external-auth-cluster-b67ng8' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f5e380>{
msg: "failed waiting for resourcegroup=\"external-auth-cluster-b67ng8\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1FWFRFUk5BTDoyREFVVEg6MkRDTFVTVEVSOjJEQjY3Tkc4LUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'external-auth-cluster-b67ng8' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001332a50>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Correlation-Request-Id": [
"8a7a4e2c-0c5c-420e-a471-80fa19b53da3",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 12 Mar 2026 04:04:21 GMT",
],
"Content-Length": ["247"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260312T040421Z:8a7a4e2c-0c5c-420e-a471-80fa19b53da3",
],
"X-Ms-Request-Id": [
"8a7a4e2c-0c5c-420e-a471-80fa19b53da3",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: D8B73840D3704...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000d743d8>:
failed to cleanup resource group: failed waiting for resourcegroup="cluster-listing-76k4gt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1DTFVTVEVSOjJETElTVElORzoyRDc2SzRHVC1FQVNUVVMyIiwiam9iTG9jYXRpb24iOiJlYXN0dXMyIn0
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'cluster-listing-76k4gt' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0004aaa40>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"cluster-listing-76k4gt\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1DTFVTVEVSOjJETElTVElORzoyRDc2SzRHVC1FQVNUVVMyIiwiam9iTG9jYXRpb24iOiJlYXN0dXMyIn0\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'cluster-listing-76k4gt' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0004aaa20>{
msg: "failed waiting for resourcegroup=\"cluster-listing-76k4gt\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1DTFVTVEVSOjJETElTVElORzoyRDc2SzRHVC1FQVNUVVMyIiwiam9iTG9jYXRpb24iOiJlYXN0dXMyIn0\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'cluster-listing-76k4gt' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001422fc0>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"c656b4eb-b02d-4689-9e61-ae252c2c4c34",
],
"X-Ms-Correlation-Request-Id": [
"c656b4eb-b02d-4689-9e61-ae252c2c4c34",
],
"X-Ms-Routing-Request-Id": [
"CANADAEAST:20260312T011924Z:c656b4eb-b02d-4689-9e61-ae252c2c4c34",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Msedge-Ref": [
"Ref A: AAC5210B403E49FB999F61B4009EE151 Ref B: CHI261140501034 Ref C: 2026-03-12T01:19:24Z",
],
"Content-Length": ["241"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Rateli...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc000d743d8>:
failed to cleanup resource group: failed waiting for resourcegroup="cluster-listing-76k4gt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1DTFVTVEVSOjJETElTVElORzoyRDc2SzRHVC1FQVNUVVMyIiwiam9iTG9jYXRpb24iOiJlYXN0dXMyIn0
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'cluster-listing-76k4gt' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0004aaa40>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"cluster-listing-76k4gt\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1DTFVTVEVSOjJETElTVElORzoyRDc2SzRHVC1FQVNUVVMyIiwiam9iTG9jYXRpb24iOiJlYXN0dXMyIn0\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'cluster-listing-76k4gt' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0004aaa20>{
msg: "failed waiting for resourcegroup=\"cluster-listing-76k4gt\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1DTFVTVEVSOjJETElTVElORzoyRDc2SzRHVC1FQVNUVVMyIiwiam9iTG9jYXRpb24iOiJlYXN0dXMyIn0\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'cluster-listing-76k4gt' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001422fc0>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"c656b4eb-b02d-4689-9e61-ae252c2c4c34",
],
"X-Ms-Correlation-Request-Id": [
"c656b4eb-b02d-4689-9e61-ae252c2c4c34",
],
"X-Ms-Routing-Request-Id": [
"CANADAEAST:20260312T011924Z:c656b4eb-b02d-4689-9e61-ae252c2c4c34",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Msedge-Ref": [
"Ref A: AAC5210B403E49FB999F61B4009EE151 Ref B: CHI261140501034 Ref C: 2026-03-12T01:19:24Z",
],
"Content-Length": ["241"],
"Content-Type": [
"application/json; charset=utf-8",
],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Rateli...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0017da1e0>:
failed to cleanup resource group: failed waiting for resourcegroup="np-autoscaling-6s9m8d" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJENlM5TThELUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'np-autoscaling-6s9m8d' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0008f0640>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"np-autoscaling-6s9m8d\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJENlM5TThELUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-6s9m8d' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008f0620>{
msg: "failed waiting for resourcegroup=\"np-autoscaling-6s9m8d\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJENlM5TThELUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-6s9m8d' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00179b020>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Request-Id": [
"cea47c2d-2981-4421-823b-471c674aec92",
],
"X-Msedge-Ref": [
"Ref A: B01DBF5715B64D88B52211B87B87B981 Ref B: CHI261140503054 Ref C: 2026-03-12T00:03:45Z",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260312T000346Z:cea47c2d-2981-4421-823b-471c674aec92",
],
"Date": [
"Thu, 12 Mar 2026 00:03:45 GMT",
],
"Content-Length": ["240"],
"Content-Type": [
"application/json; charset=utf-8",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
<*errors.joinError | 0xc0017da1e0>:
failed to cleanup resource group: failed waiting for resourcegroup="np-autoscaling-6s9m8d" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJENlM5TThELUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ResourceGroupDeletionFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupDeletionFailed",
"message": "The deletion of resource group 'np-autoscaling-6s9m8d' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details."
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0008f0640>{
msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"np-autoscaling-6s9m8d\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJENlM5TThELUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-6s9m8d' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008f0620>{
msg: "failed waiting for resourcegroup=\"np-autoscaling-6s9m8d\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1OUDoyREFVVE9TQ0FMSU5HOjJENlM5TThELUVBU1RVUzIiLCJqb2JMb2NhdGlvbiI6ImVhc3R1czIifQ\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: ResourceGroupDeletionFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceGroupDeletionFailed\",\n \"message\": \"The deletion of resource group 'np-autoscaling-6s9m8d' did not finish within allowed time. Resource group provisioning state rolled back. Please check audit logs for more details.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00179b020>{
ErrorCode: "ResourceGroupDeletionFailed",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Request-Id": [
"cea47c2d-2981-4421-823b-471c674aec92",
],
"X-Msedge-Ref": [
"Ref A: B01DBF5715B64D88B52211B87B87B981 Ref B: CHI261140503054 Ref C: 2026-03-12T00:03:45Z",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260312T000346Z:cea47c2d-2981-4421-823b-471c674aec92",
],
"Date": [
"Thu, 12 Mar 2026 00:03:45 GMT",
],
"Content-Length": ["240"],
"Content-Type": [
"application/json; charset=utf-8",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail code DenyAssignmentLimitExceeded; detail message Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at S...full failure pattern: ERROR CODE: InternalServerError; detail code DenyAssignmentLimitExceeded; detail message Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/<subscription> with search direction ScopeAndBelow. The limit is 2000.; provider Microsoft.Authorization Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 5.00%2 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 2 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapError | 0xc000579080>:
failed to create HCP cluster list-test-cluster-bq97s7: failed waiting for cluster="list-test-cluster-bq97s7" in resourcegroup="cluster-listing-8872x6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d",
"name": "46ace568-dc35-4c4f-a79f-7a406de3e15d",
"status": "Failed",
"startTime": "2026-03-11T21:08:47.154344146Z",
"endTime": "2026-03-11T21:11:43.134589726Z",
"error": {
"code": "InternalServerError",
"message": "PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/e663d1f0-257c-5415-abc3-7d8e5bb59e36\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/fcc1607d-beb4-5621-906c-726f36422d8f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/4248f691-af43-5fc6-a0fc-cbd9ca13f47c\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/16f02e63-3112-56ed-ab9e-b36ab322b94f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/288c454f-fa20-5a8b-9030-d232fc0074e7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/759088cd-864a-5f85-a334-2d9f3586f3fe\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/697e01a8-0c71-5420-b5d7-b53e6ad28d24\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/abb26984-e70f-55ad-a446-57d39d785f3a\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/077c8d44-0b2f-5e41-92d6-dc648b293341\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/895f6214-21ec-57d1-85d4-e824b8d4a5cd\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/97a2ea1b-62a8-572b-80f0-cd3287f6deae\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/bcbac9ff-fd4b-57bd-81c1-11bd09c72bc0\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/3af59b2e-b060-5c22-b9cd-9098bcb77878\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/30cb8efd-9d43-5c50-9daa-9375ce14fe98\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster list-test-cluster-bq97s7: failed waiting for cluster=\"list-test-cluster-bq97s7\" in resourcegroup=\"cluster-listing-8872x6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d\",\n \"name\": \"46ace568-dc35-4c4f-a79f-7a406de3e15d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T21:08:47.154344146Z\",\n \"endTime\": \"2026-03-11T21:11:43.134589726Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/e663d1f0-257c-5415-abc3-7d8e5bb59e36\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/fcc1607d-beb4-5621-906c-726f36422d8f\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/4248f691-af43-5fc6-a0fc-cbd9ca13f47c\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/16f02e63-3112-56ed-ab9e-b36ab322b94f\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Ba...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapError | 0xc000579080>:
failed to create HCP cluster list-test-cluster-bq97s7: failed waiting for cluster="list-test-cluster-bq97s7" in resourcegroup="cluster-listing-8872x6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d",
"name": "46ace568-dc35-4c4f-a79f-7a406de3e15d",
"status": "Failed",
"startTime": "2026-03-11T21:08:47.154344146Z",
"endTime": "2026-03-11T21:11:43.134589726Z",
"error": {
"code": "InternalServerError",
"message": "PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/e663d1f0-257c-5415-abc3-7d8e5bb59e36\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/fcc1607d-beb4-5621-906c-726f36422d8f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/4248f691-af43-5fc6-a0fc-cbd9ca13f47c\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/16f02e63-3112-56ed-ab9e-b36ab322b94f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/288c454f-fa20-5a8b-9030-d232fc0074e7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/759088cd-864a-5f85-a334-2d9f3586f3fe\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/697e01a8-0c71-5420-b5d7-b53e6ad28d24\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/abb26984-e70f-55ad-a446-57d39d785f3a\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/077c8d44-0b2f-5e41-92d6-dc648b293341\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/895f6214-21ec-57d1-85d4-e824b8d4a5cd\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/97a2ea1b-62a8-572b-80f0-cd3287f6deae\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/bcbac9ff-fd4b-57bd-81c1-11bd09c72bc0\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/3af59b2e-b060-5c22-b9cd-9098bcb77878\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/30cb8efd-9d43-5c50-9daa-9375ce14fe98\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster list-test-cluster-bq97s7: failed waiting for cluster=\"list-test-cluster-bq97s7\" in resourcegroup=\"cluster-listing-8872x6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/46ace568-dc35-4c4f-a79f-7a406de3e15d\",\n \"name\": \"46ace568-dc35-4c4f-a79f-7a406de3e15d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T21:08:47.154344146Z\",\n \"endTime\": \"2026-03-11T21:11:43.134589726Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/e663d1f0-257c-5415-abc3-7d8e5bb59e36\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/fcc1607d-beb4-5621-906c-726f36422d8f\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/4248f691-af43-5fc6-a0fc-cbd9ca13f47c\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-8872x6-managed/providers/Microsoft.Authorization/denyAssignments/16f02e63-3112-56ed-ab9e-b36ab322b94f\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Ba...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapError | 0xc00097c280>:
failed to create HCP cluster list-test-cluster-jpcgwk: failed waiting for cluster="list-test-cluster-jpcgwk" in resourcegroup="cluster-listing-2jgp6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920",
"name": "625b1b18-4c75-473a-934a-a164ea9e7920",
"status": "Failed",
"startTime": "2026-03-11T21:08:59.912972248Z",
"endTime": "2026-03-11T21:11:11.150064729Z",
"error": {
"code": "InternalServerError",
"message": "PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/c830eb67-f4cc-5c33-a275-01641a3773ec\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/439dd205-e252-5fd3-8924-58746f329066\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/faa2951f-b181-51b3-a546-c3af6657963e\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster list-test-cluster-jpcgwk: failed waiting for cluster=\"list-test-cluster-jpcgwk\" in resourcegroup=\"cluster-listing-2jgp6g\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920\",\n \"name\": \"625b1b18-4c75-473a-934a-a164ea9e7920\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T21:08:59.912972248Z\",\n \"endTime\": \"2026-03-11T21:11:11.150064729Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/c830eb67-f4cc-5c33-a275-01641a3773ec\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/439dd205-e252-5fd3-8924-58746f329066\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/faa2951f-b181-51b3-a546-c3af6657963e\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00097c260>{
msg: "failed waiting for cluster=\"list-test-cluster-jpcgwk\" in resourcegroup=\"cluster-listing-2jgp6g\" to finish creating: GET https://management.azure.com/subscriptio...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
<*fmt.wrapError | 0xc00097c280>:
failed to create HCP cluster list-test-cluster-jpcgwk: failed waiting for cluster="list-test-cluster-jpcgwk" in resourcegroup="cluster-listing-2jgp6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920",
"name": "625b1b18-4c75-473a-934a-a164ea9e7920",
"status": "Failed",
"startTime": "2026-03-11T21:08:59.912972248Z",
"endTime": "2026-03-11T21:11:11.150064729Z",
"error": {
"code": "InternalServerError",
"message": "PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/c830eb67-f4cc-5c33-a275-01641a3773ec\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/439dd205-e252-5fd3-8924-58746f329066\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/faa2951f-b181-51b3-a546-c3af6657963e\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: DenyAssignmentLimitExceeded\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DenyAssignmentLimitExceeded\",\n \"message\": \"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster list-test-cluster-jpcgwk: failed waiting for cluster=\"list-test-cluster-jpcgwk\" in resourcegroup=\"cluster-listing-2jgp6g\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/625b1b18-4c75-473a-934a-a164ea9e7920\",\n \"name\": \"625b1b18-4c75-473a-934a-a164ea9e7920\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-11T21:08:59.912972248Z\",\n \"endTime\": \"2026-03-11T21:11:11.150064729Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/c830eb67-f4cc-5c33-a275-01641a3773ec\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/439dd205-e252-5fd3-8924-58746f329066\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\\nPUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-2jgp6g-managed/providers/Microsoft.Authorization/denyAssignments/faa2951f-b181-51b3-a546-c3af6657963e\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: DenyAssignmentLimitExceeded\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DenyAssignmentLimitExceeded\\\",\\n \\\"message\\\": \\\"Cannot add a new DenyAssignment to the store. There are already 2000 deny assignments at Scope=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with search direction ScopeAndBelow. The limit is 2000.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00097c260>{
msg: "failed waiting for cluster=\"list-test-cluster-jpcgwk\" in resourcegroup=\"cluster-listing-2jgp6g\" to finish creating: GET https://management.azure.com/subscriptio...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 5.00%2 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 2 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0016f80f0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-qks87c" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00073ddc0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-qks87c\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00073dda0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-qks87c\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00073d8e0>{
msg: "failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-qks87c\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0680c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0016f80f0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-qks87c" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00073ddc0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-qks87c\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00073dda0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-qks87c\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00073d8e0>{
msg: "failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-qks87c\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0680c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000fb62e8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-qb9hch" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00095f2a0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-qb9hch\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00095f280>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-qb9hch\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00095f260>{
msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-qb9hch\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0680c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000fb62e8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-qb9hch" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00095f2a0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-qb9hch\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00095f280>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-qb9hch\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00095f260>{
msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-qb9hch\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb0680c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||
Should receive valid HTTP status code from authorized IPfull failure pattern: Should receive valid HTTP status code from authorized IP Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.50%1 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | INT | |||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:167 with:
Should receive valid HTTP status code from authorized IP
Expected
<string>: 000
to match regular expression
<string>: ^[2-5][0-9][0-9]$
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:167 with:
Should receive valid HTTP status code from authorized IP
Expected
<string>: 000
to match regular expression
<string>: ^[2-5][0-9][0-9]$Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4full failure pattern: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.50%1 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 1 · Mar 14: 0 · Mar 15: 0 | INT, STG | |||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.006s.
Expected success, but got an error:
<*errors.joinError | 0xc000f195d8>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc000cbc700>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0008f4540>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.006s.
Expected success, but got an error:
<*errors.joinError | 0xc000f195d8>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
{
errs: [
<*fmt.wrapError | 0xc000cbc700>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
err: <*errors.errorString | 0xc0008f4540>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
},
},
],
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||
failed to create route: the server is currently unable to handle the requestfull failure pattern: failed to create route: the server is currently unable to handle the request Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.50%1 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 0 · Mar 13: 1 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:131]: Unexpected error:
<*fmt.wrapError | 0xc0009bfcc0>:
failed to create route: the server is currently unable to handle the request
{
msg: "failed to create route: the server is currently unable to handle the request",
err: <*errors.StatusError | 0xc000f1fe00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "the server is currently unable to handle the request",
Reason: "ServiceUnavailable",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "error trying to reach service: tls: first record does not look like a TLS handshake",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 503,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:131]: Unexpected error:
<*fmt.wrapError | 0xc0009bfcc0>:
failed to create route: the server is currently unable to handle the request
{
msg: "failed to create route: the server is currently unable to handle the request",
err: <*errors.StatusError | 0xc000f1fe00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "the server is currently unable to handle the request",
Reason: "ServiceUnavailable",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "error trying to reach service: tls: first record does not look like a TLS handshake",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 503,
},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||
or adjust the parameters in Gomega's 'format' package.full failure pattern: or adjust the parameters in Gomega's 'format' package. Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.50%1 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc0013df1a0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "Cluster '2ov6s0cpl7gkpuue1b516qhk8cdsmabo' not found"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "NotFound",
StatusCode: 404,
RawResponse: {
Status: "404 Not Found",
StatusCode: 404,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["126"],
"X-Envoy-Upstream-Service-Time": ["584"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
"CANADAEAST:20260311T013040Z:c842b23a-1dd5-473c-a571-3d4df49b88a4",
],
"X-Msedge-Ref": [
"Ref A: 0090BB3CE36E41728AA688E9FBBE247F Ref B: CHI261140503023 Ref C: 2026-03-11T01:30:39Z",
],
"X-Ms-Error-Code": ["NotFound"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Correlation-Request-Id": [
"c842b23a-1dd5-473c-a571-3d4df49b88a4",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 11 Mar 2026 01:30:40 GMT",
],
"X-Ms-Request-Id": [
"7385551f-2b14-46ef-b16b-b7df9d55e6e4",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001484b80>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 78, 111, 116, 70, 111, 117, 110, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 111, 118, 54, 115, 48, 99, 112, 108, 55, 103, 107, 112, 117, 117, 101, 49, 98, 53, 49, 54, 113, 104, 107, 56, 99, 100, 115, 109, 97, 98, 111, 39, 32, 110, 111, 116, 32, 102, 111, 117, 110, 100, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 126,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:97]: Expected
<*exported.ResponseError | 0xc0013df1a0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "Cluster '2ov6s0cpl7gkpuue1b516qhk8cdsmabo' not found"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "NotFound",
StatusCode: 404,
RawResponse: {
Status: "404 Not Found",
StatusCode: 404,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["126"],
"X-Envoy-Upstream-Service-Time": ["584"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
"CANADAEAST:20260311T013040Z:c842b23a-1dd5-473c-a571-3d4df49b88a4",
],
"X-Msedge-Ref": [
"Ref A: 0090BB3CE36E41728AA688E9FBBE247F Ref B: CHI261140503023 Ref C: 2026-03-11T01:30:39Z",
],
"X-Ms-Error-Code": ["NotFound"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Correlation-Request-Id": [
"c842b23a-1dd5-473c-a571-3d4df49b88a4",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Wed, 11 Mar 2026 01:30:40 GMT",
],
"X-Ms-Request-Id": [
"7385551f-2b14-46ef-b16b-b7df9d55e6e4",
],
},
Body: <*exported.nopClosingBytesReader | 0xc001484b80>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 78, 111, 116, 70, 111, 117, 110, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 111, 118, 54, 115, 48, 99, 112, 108, 55, 103, 107, 112, 117, 117, 101, 49, 98, 53, 49, 54, 113, 104, 107, 56, 99, 100, 115, 109, 97, 98, 111, 39, 32, 110, 111, 116, 32, 102, 111, 117, 110, 100, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 126,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
],
"Authorization": ["redacted"],
},
Body: nil,
GetBody: nil,
ContentLength: 0,
TransferEncoding: nil,
Close: false,
Host: "management.azure.com",
Form: nil,
PostForm: nil,
MultipartForm: nil,
Trailer: nil,
RemoteAddr: "",
RequestURI: "",
TLS: nil,
Cancel: nil,
Response: nil,
Pattern: "",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
to be nilContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||
route host was never found: Get "<url>": dial tcp <ip>:<port>: i/o timeoutfull failure pattern: route host was never found: Get "<url>": dial tcp <ip>:<port>: i/o timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.50%1 of 40 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 1 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:131]: Unexpected error:
<*fmt.wrapError | 0xc0008ac140>:
route host was never found: Get "https://agnhost-e2e-serving-app-qm5jk.apps.aro.a5a4c4g0z8w2g8w.e8s3.canadacentral.aroapp-hcp.io": dial tcp 20.116.240.254:443: i/o timeout
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-qm5jk.apps.aro.a5a4c4g0z8w2g8w.e8s3.canadacentral.aroapp-hcp.io\": dial tcp 20.116.240.254:443: i/o timeout",
err: <*url.Error | 0xc000f84300>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-qm5jk.apps.aro.a5a4c4g0z8w2g8w.e8s3.canadacentral.aroapp-hcp.io",
Err: <*net.OpError | 0xc000cb8320>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: <*net.TCPAddr | 0xc000f842d0>{
IP: [20, 116, 240, 254],
Port: 443,
Zone: "",
},
Err: <*net.timeoutError | 0xb35be00>{},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:131]: Unexpected error:
<*fmt.wrapError | 0xc0008ac140>:
route host was never found: Get "https://agnhost-e2e-serving-app-qm5jk.apps.aro.a5a4c4g0z8w2g8w.e8s3.canadacentral.aroapp-hcp.io": dial tcp 20.116.240.254:443: i/o timeout
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-qm5jk.apps.aro.a5a4c4g0z8w2g8w.e8s3.canadacentral.aroapp-hcp.io\": dial tcp 20.116.240.254:443: i/o timeout",
err: <*url.Error | 0xc000f84300>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-qm5jk.apps.aro.a5a4c4g0z8w2g8w.e8s3.canadacentral.aroapp-hcp.io",
Err: <*net.OpError | 0xc000cb8320>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: <*net.TCPAddr | 0xc000f842d0>{
IP: [20, 116, 240, 254],
Port: 443,
Zone: "",
},
Err: <*net.timeoutError | 0xb35be00>{},
},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||