Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 144 | 63.89% (off track) | -31.11pp | +3.67pp | 93.08% (121/130) | +10.22pp | 76.03% (92/121) | -1.21pp |
| INT | 95% - All E2E job runs | 42 | 47.62% (off track) | -47.38pp | +0.95pp | n/a | n/a | 47.62% (20/42) | +0.95pp |
| STG | 95% - All E2E job runs | 19 | 42.11% (off track) | -52.89pp | -40.25pp | n/a | n/a | 44.44% (8/18) | -37.91pp |
| PROD | 95% - All E2E job runs | 43 | 53.49% (off track) | -41.51pp | +4.77pp | n/a | n/a | 56.10% (23/41) | +6.10pp |
Environment: DEV
E2E Jobs
375
Success Rate
47.20%
Provision success
Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.80.49% (264/328)
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.67.05% (177/264)
E2E Jobs (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the number of runs in that DEV-only goal basis.144
Success Rate (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the overall success rate for that DEV-only goal basis.63.89%
Provision success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.93.08% (121/130)
E2E success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.76.03% (92/121)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
Successful runs (post-good + batches)
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 73.50% | 117 | rp-api-compat-all/parallel | Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 | |
| 84.62% | 52 | rp-api-compat-all/parallel | Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API | |
| 93.80% | 258 | rp-api-compat-all/parallel | Customer should be able to update nodepool replicas and autoscaling | |
| 94.61% | 241 | rp-api-compat-all/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 94.96% | 258 | rp-api-compat-all/parallel | Customer should be able to update node pool labels and taints |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 28 | 7.47%28 of 375 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 15 · Mar 25: 13 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc001048640>:
failed to create HCP cluster cluster-zstream-4-19-qr8gpv: failed starting cluster creation "cluster-zstream-4-19-qr8gpv" in resourcegroup="rg-zstream-upgrade-4-19-nkf9gt": PUT https://rp.j2325760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nkf9gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-qr8gpv
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-qr8gpv: failed starting cluster creation \"cluster-zstream-4-19-qr8gpv\" in resourcegroup=\"rg-zstream-upgrade-4-19-nkf9gt\": PUT https://rp.j2325760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nkf9gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-qr8gpv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001048620>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-qr8gpv\" in resourcegroup=\"rg-zstream-upgrade-4-19-nkf9gt\": PUT https://rp.j2325760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nkf9gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-qr8gpv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000d88060>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"1cefaa67-fd64-4421-828d-46caba74490c",
],
"Date": [
"Wed, 25 Mar 2026 07:58:14 GMT",
],
"Content-Length": ["138"],
"X-Envoy-Upstream-Service-Time": ["62"],
},
Body: <*exported.nopClosingBytesReader | 0xc001048560>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 138,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc001048640>:
failed to create HCP cluster cluster-zstream-4-19-qr8gpv: failed starting cluster creation "cluster-zstream-4-19-qr8gpv" in resourcegroup="rg-zstream-upgrade-4-19-nkf9gt": PUT https://rp.j2325760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nkf9gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-qr8gpv
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-qr8gpv: failed starting cluster creation \"cluster-zstream-4-19-qr8gpv\" in resourcegroup=\"rg-zstream-upgrade-4-19-nkf9gt\": PUT https://rp.j2325760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nkf9gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-qr8gpv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001048620>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-qr8gpv\" in resourcegroup=\"rg-zstream-upgrade-4-19-nkf9gt\": PUT https://rp.j2325760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nkf9gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-qr8gpv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000d88060>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"1cefaa67-fd64-4421-828d-46caba74490c",
],
"Date": [
"Wed, 25 Mar 2026 07:58:14 GMT",
],
"Content-Length": ["138"],
"X-Envoy-Upstream-Service-Time": ["62"],
},
Body: <*exported.nopClosingBytesReader | 0xc001048560>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 138,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: Unexpected error:
<*fmt.wrapError | 0xc000f411c0>:
failed to create HCP cluster cluster-zstream-4-19-nnjrbj: failed starting cluster creation "cluster-zstream-4-19-nnjrbj" in resourcegroup="rg-zstream-upgrade-4-19-vgbb4l": PUT https://rp.j9577728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vgbb4l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nnjrbj
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-nnjrbj: failed starting cluster creation \"cluster-zstream-4-19-nnjrbj\" in resourcegroup=\"rg-zstream-upgrade-4-19-vgbb4l\": PUT https://rp.j9577728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vgbb4l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nnjrbj\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f411a0>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-nnjrbj\" in resourcegroup=\"rg-zstream-upgrade-4-19-vgbb4l\": PUT https://rp.j9577728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vgbb4l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nnjrbj\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001717830>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["49"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"6d7882ca-03eb-4641-84b4-c8342a8cdd86",
],
"Date": [
"Wed, 25 Mar 2026 07:29:36 GMT",
],
"Content-Length": ["138"],
},
Body: <*exported.nopClosingBytesReader | 0xc000f410e0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 138,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: Unexpected error:
<*fmt.wrapError | 0xc000f411c0>:
failed to create HCP cluster cluster-zstream-4-19-nnjrbj: failed starting cluster creation "cluster-zstream-4-19-nnjrbj" in resourcegroup="rg-zstream-upgrade-4-19-vgbb4l": PUT https://rp.j9577728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vgbb4l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nnjrbj
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-nnjrbj: failed starting cluster creation \"cluster-zstream-4-19-nnjrbj\" in resourcegroup=\"rg-zstream-upgrade-4-19-vgbb4l\": PUT https://rp.j9577728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vgbb4l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nnjrbj\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000f411a0>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-nnjrbj\" in resourcegroup=\"rg-zstream-upgrade-4-19-vgbb4l\": PUT https://rp.j9577728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vgbb4l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nnjrbj\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001717830>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["49"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"6d7882ca-03eb-4641-84b4-c8342a8cdd86",
],
"Date": [
"Wed, 25 Mar 2026 07:29:36 GMT",
],
"Content-Length": ["138"],
},
Body: <*exported.nopClosingBytesReader | 0xc000f410e0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 138,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc00100a600>:
failed to create HCP cluster cluster-zstream-4-19-thhk92: failed starting cluster creation "cluster-zstream-4-19-thhk92" in resourcegroup="rg-zstream-upgrade-4-19-kp9n2g": PUT https://rp.j7794048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-kp9n2g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-thhk92
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-thhk92: failed starting cluster creation \"cluster-zstream-4-19-thhk92\" in resourcegroup=\"rg-zstream-upgrade-4-19-kp9n2g\": PUT https://rp.j7794048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-kp9n2g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-thhk92\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00100a5e0>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-thhk92\" in resourcegroup=\"rg-zstream-upgrade-4-19-kp9n2g\": PUT https://rp.j7794048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-kp9n2g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-thhk92\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0010b2420>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["138"],
"X-Envoy-Upstream-Service-Time": ["55"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"904d0344-5172-446f-bf34-9aeb5b14a238",
],
"Date": [
"Wed, 25 Mar 2026 06:06:48 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00100a520>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 138,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc00100a600>:
failed to create HCP cluster cluster-zstream-4-19-thhk92: failed starting cluster creation "cluster-zstream-4-19-thhk92" in resourcegroup="rg-zstream-upgrade-4-19-kp9n2g": PUT https://rp.j7794048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-kp9n2g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-thhk92
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-thhk92: failed starting cluster creation \"cluster-zstream-4-19-thhk92\" in resourcegroup=\"rg-zstream-upgrade-4-19-kp9n2g\": PUT https://rp.j7794048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-kp9n2g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-thhk92\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00100a5e0>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-thhk92\" in resourcegroup=\"rg-zstream-upgrade-4-19-kp9n2g\": PUT https://rp.j7794048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-kp9n2g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-thhk92\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0010b2420>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["138"],
"X-Envoy-Upstream-Service-Time": ["55"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"904d0344-5172-446f-bf34-9aeb5b14a238",
],
"Date": [
"Wed, 25 Mar 2026 06:06:48 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00100a520>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 138,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (28) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | e2e | 18 | 4.80%18 of 375 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsMar 22: 0 · Mar 23: 1 · Mar 24: 7 · Mar 25: 11 · Mar 26: 0 · Mar 27: 1 · Mar 28: 0 · Mar 29: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc000a293e0>:
failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-4h255n" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4h255n, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-4h255n\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4h255n, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000095810>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4h255n",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc000a293e0>:
failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-4h255n" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4h255n, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-4h255n\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4h255n, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000095810>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4h255n",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc001095140>:
failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kgkt2t" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kgkt2t, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-kgkt2t\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kgkt2t, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015f3c10>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kgkt2t",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc001095140>:
failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kgkt2t" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kgkt2t, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-kgkt2t\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kgkt2t, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015f3c10>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kgkt2t",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc0014ecea0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-8mrp5r" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8mrp5r, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-8mrp5r\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8mrp5r, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000c8f780>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8mrp5r",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc0014ecea0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-8mrp5r" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8mrp5r, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-8mrp5r\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8mrp5r, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000c8f780>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8mrp5r",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (18) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 11 | 2.93%11 of 375 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 21 · Mar 25: 7 · Mar 26: 3 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | INT, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc001444f90>:
failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2nbkt4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<*fmt.wrapErrors | 0xc001444f60>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc001444f90>:
failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2nbkt4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<*fmt.wrapErrors | 0xc001444f60>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc0011d91a0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-48kgkn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-48kgkn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001196aa0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc0011d91a0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-48kgkn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-48kgkn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001196aa0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc001287800>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cndfg8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-cndfg8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00140f540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc001287800>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cndfg8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-cndfg8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00140f540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (19)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
make[2]: *** [Makefile:32: deploy] Error 1full failure pattern: make[2]: *** [Makefile:32: deploy] Error 1 Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | provision | 10 | 2.67%10 of 375 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsMar 22: 0 · Mar 23: 8 · Mar 24: 1 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-25T01:33:03.092Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-25T01:43:24.338Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=c2e7341b-512a-449d-8565-627b037718c6 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=d0619a55-a6f4-4abc-ae90-3ae1e8d776ea && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj6890368.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj6890368.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j6890368 -n arohcp-prow-dbcs-j6890368 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6890368/providers/Microsoft.Network/dnszones/j6890368.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j6890368-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j6890368-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:5f079a0240687e6d2214d892a61019db3a3bbb4e3afb97bcf7731633af4ce7d8 \\\n --set global.imageDigest=sha256:5f079a0240687e6d2214d892a61019db3a3bbb4e3afb97bcf7731633af4ce7d8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j6890368-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"73f0671f-7b25-4bb0-bddc-c533574a227c\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j6890368-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6890368-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j6890368-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: context deadline exceeded\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"time=2026-03-24T23:09:24.452Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-24T23:19:44.737Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=bba9ecaf-0a5e-47af-8d23-c33286ffeb31 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=f3698ec3-5fdb-4fee-8a48-711baac420de && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3852928.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3852928.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3852928 -n arohcp-prow-dbcs-j3852928 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3852928/providers/Microsoft.Network/dnszones/j3852928.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3852928-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3852928-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:5f079a0240687e6d2214d892a61019db3a3bbb4e3afb97bcf7731633af4ce7d8 \\\n --set global.imageDigest=sha256:5f079a0240687e6d2214d892a61019db3a3bbb4e3afb97bcf7731633af4ce7d8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j3852928-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"9a47fa51-2376-4a6c-b822-670de4dd981b\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3852928-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3852928-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3852928-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: context deadline exceeded\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"time=2026-03-23T21:43:19.090Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n"
time=2026-03-23T21:43:37.165Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=a4d3a2a1-c8bb-4792-8cae-355b5030e10c && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=bf8ce325-899b-4289-8318-606037f6dfde && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj2975104.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj2975104.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j2975104 -n arohcp-prow-dbcs-j2975104 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2975104/providers/Microsoft.Network/dnszones/j2975104.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j2975104-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j2975104-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set global.imageDigest=sha256:e30ee0c2851431cc17f44a109f987cb845e77369036c9a8074f17b66163341c8 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j2975104-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"51a31a28-8560-4ccd-b603-6213d8bf44ac\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j2975104-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2975104-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j2975104-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: create: failed to create: Unauthorized\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGridfull failure pattern: ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | provision | 8 | 2.13%8 of 375 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 4 · Mar 26: 1 · Mar 27: 3 · Mar 28: 0 · Mar 29: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-27T15:06:36.372Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-27T15:06:38.469Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-27T15:06:39.655Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=6b04bb802b99afb35d3a7da01b435a2fe54542974d8c55f4de7c0fd1253cb1ef
time=2026-03-27T15:12:11.166Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8121216/providers/Microsoft.Resources/deployments/6b04bb802b99afb35d3a7da01b435a2fe54542974d8c55f4de7c0fd1253cb1ef/operationStatuses/08584269828865178334\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8121216/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8121216/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j8121216\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report cdabde50-a0ae-4404-900f-94278705eafb:3/27/2026 3:09:17 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-03-27T13:50:41.577Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-27T13:50:43.752Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-27T13:50:45.176Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=6be2ba751dc97e52c6d9bfdb5b511a47d6811cd308995a068a3f14585b18f93b
time=2026-03-27T13:53:46.117Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6628864/providers/Microsoft.Resources/deployments/6be2ba751dc97e52c6d9bfdb5b511a47d6811cd308995a068a3f14585b18f93b/operationStatuses/08584269874412386907\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6628864/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6628864/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j6628864\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 39fe6ca5-272a-4487-8e34-a1281db54110:3/27/2026 1:52:06 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-03-27T12:12:30.979Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-27T12:12:33.010Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-27T12:12:34.380Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=5ad762b00291321f96cc50f58624231bd6cdcebe8741580f3f82f1cf16401693
time=2026-03-27T12:15:05.312Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3974784/providers/Microsoft.Resources/deployments/5ad762b00291321f96cc50f58624231bd6cdcebe8741580f3f82f1cf16401693/operationStatuses/08584269933320118051\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3974784/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3974784/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j3974784\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 9f26347b-934c-4bcd-a70a-9dbc2b6b0253:3/27/2026 12:14:54 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4full failure pattern: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 6 | 1.60%6 of 375 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 2 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 8 · Mar 29: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.009s.
Expected success, but got an error:
<*errors.joinError | 0xc00101c4e0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
{
errs: [
<*fmt.wrapError | 0xc000b00c20>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
err: <*errors.errorString | 0xc000804560>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.009s.
Expected success, but got an error:
<*errors.joinError | 0xc00101c4e0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
{
errs: [
<*fmt.wrapError | 0xc000b00c20>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
err: <*errors.errorString | 0xc000804560>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.001s.
Expected success, but got an error:
<*errors.joinError | 0xc00100f2f0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
{
errs: [
<*fmt.wrapError | 0xc000956ee0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
err: <*errors.errorString | 0xc0012f65e0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.001s.
Expected success, but got an error:
<*errors.joinError | 0xc00100f2f0>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
{
errs: [
<*fmt.wrapError | 0xc000956ee0>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
err: <*errors.errorString | 0xc0012f65e0>{
s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.007s.
Expected success, but got an error:
<*errors.joinError | 0xc0013a6570>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
{
errs: [
<*fmt.wrapError | 0xc001449840>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
err: <*errors.errorString | 0xc0007bce30>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.007s.
Expected success, but got an error:
<*errors.joinError | 0xc0013a6570>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
{
errs: [
<*fmt.wrapError | 0xc001449840>{
msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
err: <*errors.errorString | 0xc0007bce30>{
s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
},
},
],
}Contributing tests (2)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>.full failure pattern: ERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>. Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | provision | 5 | 1.33%5 of 375 job runs affected | RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | Show trend detailsMar 22: 0 · Mar 23: 4 · Mar 24: 1 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-24T03:04:07.422Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-24T03:04:07.900Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-24T03:04:08.871Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=6f5dc51387bf3d351f1b8e5546f891a1318b5147b96d6ce634f7f9b9588dd09a
time=2026-03-24T03:04:39.110Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1837696-mgmt-1/providers/Microsoft.Resources/deployments/6f5dc51387bf3d351f1b8e5546f891a1318b5147b96d6ce634f7f9b9588dd09a/operationStatuses/08584272854372039071\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is 0890b582518a5466b9bb14be0e9debd7.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-03-24T00:31:24.809Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-24T00:31:25.578Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-24T00:31:26.601Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=7a15931e1f96af913f2cb12fb2d11d6e14f7d5aaaf1b1b54309a877efbe4f835
time=2026-03-24T00:31:56.909Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8096384-mgmt-1/providers/Microsoft.Resources/deployments/7a15931e1f96af913f2cb12fb2d11d6e14f7d5aaaf1b1b54309a877efbe4f835/operationStatuses/08584272945994994062\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is c2458f5d40a058a9b4d76c1a122ba64b.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-03-23T23:34:29.701Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n Kind: ARM\n Template: templates/svc-mgmt-permissions.bicep\n Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-23T23:34:30.234Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-23T23:34:31.142Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=1274b489e405478a98e50259476f20a0959f9f26bb68b51343591eb230be2a5e
time=2026-03-23T23:35:01.387Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9211264-mgmt-1/providers/Microsoft.Resources/deployments/1274b489e405478a98e50259476f20a0959f9f26bb68b51343591eb230be2a5e/operationStatuses/08584272980148477728\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is 37ef272e2ef651f1be383cfc74698550.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceededfull failure pattern: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | provision | 5 | 1.33%5 of 375 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsMar 22: 1 · Mar 23: 0 · Mar 24: 1 · Mar 25: 3 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-03-26T09:48:18.147Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-26T09:48:18.853Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1912448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1912448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:e9330836-104c-4ff6-8896-5e80a6561853 databaseName:arohcpprow-rp-j1912448 databaseUrl:https://arohcpprow-rp-j1912448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-icg9x6nq/pipeline@sha256:d7affa8c94e1be1641298f474a1a1409323fb682a03fba66b2735fdfe48a905a replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-icg9x6nq/pipeline:pull workloadIdentityClientId:bbe90c53-94c0-41a0-a862-22f9c8836d86 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:e9330836-104c-4ff6-8896-5e80a6561853 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-26T09:48:18.853Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:48:18.912Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-03-26T09:48:19.026Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-03-26T09:48:19.026Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:48:19.137Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:23.504Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-26T09:53:23.504Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:23.504Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:27.747Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1912448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1912448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:e9330836-104c-4ff6-8896-5e80a6561853 databaseName:arohcpprow-rp-j1912448 databaseUrl:https://arohcpprow-rp-j1912448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-icg9x6nq/pipeline@sha256:d7affa8c94e1be1641298f474a1a1409323fb682a03fba66b2735fdfe48a905a replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-icg9x6nq/pipeline:pull workloadIdentityClientId:bbe90c53-94c0-41a0-a862-22f9c8836d86 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:e9330836-104c-4ff6-8896-5e80a6561853 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-26T09:53:27.748Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4xTTW8aMRS876942hNI64hCGzVUOaRpT1Uq1PSUqgevPV0cvH6rZy+IiB9fLSRhhSDhOm9m3oc9mUciQeRWDCJdk9VJJ116DLh8hEk/XLDTmMSFqqAd9FPXeIWwRNgisdHmBR7+yYiI8lsO/1x1p5u8oLzUZoFglX5qBUrakFwNZbaUrq6F1dw0efGG9iT7F3t0sIe2EAUPkxyHE8yvLlgXqr7z+8JvaDyva4TUK6pn/RH+jZFZ6/2RXk3rvSr3+IHwHrJ0BnccXGI5r9mz5sYYbncDvss9YqxqJHEmHtumTXMW96S768zYO7Pesrzn1YvsjflmgHQWCMkZfXDhM9rfwwjSTHjpLOTW6xj7F427ldQCa7XUre8/UZ79/ZIt2hISkBC/dx82ZhtazSGg7hPGpOuGSqQVEGhgdUIHD/LxaHypRhM1vvw9upp+/Dz9cPWQD+nigk5xPk2m48lDPhxmG3pkF2jhgr12IUB6MeNA+3T1Y3WYp2xDjXBX3Q9aUMN2Rz7PpCDPVUE1YtRVZ8liIVSue8tbRPM/AAD//3SFvhEMBAAA" time=2026-03-26T09:53:27.751Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:27.751Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-25T18:09:25.661Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-25T18:09:26.326Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj5199488.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj5199488.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:88887d6c-e73b-4143-8dc2-0939f3daf643 databaseName:arohcpprow-rp-j5199488 databaseUrl:https://arohcpprow-rp-j5199488.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-yji9fhnx/pipeline@sha256:d6d335399aff92b8825e3d7b0e40e3f3315d0b5255ba48b825aa9a56a19bf4df replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-yji9fhnx/pipeline:pull workloadIdentityClientId:09ddc654-76dc-4eda-88e3-ff98e3106c84 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:88887d6c-e73b-4143-8dc2-0939f3daf643 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-25T18:09:26.326Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T18:09:26.519Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-03-25T18:09:26.755Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-03-25T18:09:26.755Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T18:09:26.872Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T18:14:34.005Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-25T18:14:34.005Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T18:14:34.005Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T18:14:39.352Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj5199488.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj5199488.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:88887d6c-e73b-4143-8dc2-0939f3daf643 databaseName:arohcpprow-rp-j5199488 databaseUrl:https://arohcpprow-rp-j5199488.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-yji9fhnx/pipeline@sha256:d6d335399aff92b8825e3d7b0e40e3f3315d0b5255ba48b825aa9a56a19bf4df replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-yji9fhnx/pipeline:pull workloadIdentityClientId:09ddc654-76dc-4eda-88e3-ff98e3106c84 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:88887d6c-e73b-4143-8dc2-0939f3daf643 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-25T18:14:39.354Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTT28aMRDF7/spRnsCaR0RkkYtVQ407alKhdqeUvXgtV/BwetZjb0gIj58tQsJKwQt1+ff/H2ezCORIHIjBpHuyeqkky49Blw+w6SvLthJTOLCvKCd9E1XeJOwQuiUWGvzKg9/ZURE+WfUnjcVQsoLyrWwWphaldosEWxPyosdPzUya7z/5IJ1Yd4Ce1bVjfeqPOhHgTNApk1aICRndHIcThRUFZI4E0/EP3D44+aPuu7X1C+NQEkTkqugTIdcGnuW/s4ereyhLUTBwxy1+7aMJi1Y3Es3zoy9M5uO8p7Xr6P8Y5c/YARpJrxyFvLgdYz9BiNk5QzUEhu10o1PJ1N0zCMHl1gus3AfMzWGm53t59l2Fye8/v9q9kUu9Dj7/TFbNiUkICF+aT9szLa0XkBArbcx6aqmEmkNBBpYndDKg3w8Gt+p0Y0av/t5/X4y+jAZ3z3lQ7q6onPM9e3k5vYpHw6zLT2zC7R0wd67ECC9M+NAh+vqn9XxPWVbqoXb10OjBdVsd/BlSQryPC+oQox63qZksRAqN73hLaL5GwAA//+m9XKXDAQAAA==" time=2026-03-25T18:14:39.355Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T18:14:39.356Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-25T11:15:27.982Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-25T11:15:28.530Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1052160.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1052160.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:c2e6ab23-ec8e-49c5-9ab3-6398f0249a78 databaseName:arohcpprow-rp-j1052160 databaseUrl:https://arohcpprow-rp-j1052160.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0h1g4mvn/pipeline@sha256:b308d5d3c40c84dd278ba5c57f26a4f1c8487c2f22221e35a07f5b42e785fd29 replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0h1g4mvn/pipeline:pull workloadIdentityClientId:993e59d4-4554-4203-a59f-77f172f9cdaa workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:c2e6ab23-ec8e-49c5-9ab3-6398f0249a78 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-25T11:15:28.531Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T11:15:28.593Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-03-25T11:15:28.716Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-03-25T11:15:28.716Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T11:15:28.834Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T11:20:34.474Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-25T11:20:34.474Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T11:20:34.474Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T11:20:38.263Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1052160.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1052160.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:c2e6ab23-ec8e-49c5-9ab3-6398f0249a78 databaseName:arohcpprow-rp-j1052160 databaseUrl:https://arohcpprow-rp-j1052160.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0h1g4mvn/pipeline@sha256:b308d5d3c40c84dd278ba5c57f26a4f1c8487c2f22221e35a07f5b42e785fd29 replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0h1g4mvn/pipeline:pull workloadIdentityClientId:993e59d4-4554-4203-a59f-77f172f9cdaa workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:c2e6ab23-ec8e-49c5-9ab3-6398f0249a78 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-25T11:20:38.266Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5STQW/aQBCF7/4VI59A8kZAmqqiyoGmPVWpUNtTqh7Wu6+wYb1jza5BRPz4yjgUi5KW3Ky335udt57JPBIJIjdiEOmWrE466dJjwOUjTPrsgp3GJC4sCuqkL7rCHwlrhL0Sa20O8vBHRkSUf4OsncHMGG5CygvKS21WCLb91MJqaeq86Ng7Dr/c4l7XPUzpp0agpAnJVVBmj1zqfZF+7qp3og6eCkmciWdMH1F73lboYpz4zvCzJi1Z3JNOjsOcvTPbPeU9bw7X/MM+B6QtgZCc2dd4XbvPGe85uMRyWctf2e8fxUNbiIKHObm4T35wwbqw6L/4/40zI/PG+zPeuvFelUf9rzxGkObCa2chd17H2HfHLq5aYavWuvH9n5RnP99nq6aEBCTET+3AxmxHmyUE1A5WTLqqqUTaAIEGVie08iCfjCZv1ehaTW6+j8fT8c108u4hH9LVFb3ETEbT6zcP+XCY7eiRXaCVC/bWhQDprRkHOm5Xf61O9ynbUS3cnh4bLahm28GXFSnI86KgCjHqRVuSxUKo3PbCW0TzOwAA///R1mERDAQAAA==" time=2026-03-25T11:20:38.269Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-25T11:20:38.269Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 5 | 1.33%5 of 375 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsMar 22: 0 · Mar 23: 11 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:125]: Unexpected error:
<*fmt.wrapError | 0xc0000dcb00>:
failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000912ff0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"b163422a-6a86-40c3-af8a-af46b3a6eb36",
],
"Date": [
"Mon, 23 Mar 2026 19:13:37 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["64"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0000dc580>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7310080.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:125]: Unexpected error:
<*fmt.wrapError | 0xc0000dcb00>:
failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://rp.j7310080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000912ff0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"b163422a-6a86-40c3-af8a-af46b3a6eb36",
],
"Date": [
"Mon, 23 Mar 2026 19:13:37 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["64"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0000dc580>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7310080.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-v25j69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2024-06-10-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"Accept": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:139]: Unexpected error:
<*fmt.wrapError | 0xc0008c0c00>:
failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-rhqc66" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"ephemeral-np\" for cluster \"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rhqc66\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000937440>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"e2e35a32-25dd-4716-bde4-6f5b2d4421c3",
],
"Date": [
"Mon, 23 Mar 2026 18:53:09 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["3006"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0008c0b40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T18:52:45Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:139]: Unexpected error:
<*fmt.wrapError | 0xc0008c0c00>:
failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-rhqc66" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for nodepool=\"ephemeral-np\" for cluster \"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rhqc66\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000937440>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"e2e35a32-25dd-4716-bde4-6f5b2d4421c3",
],
"Date": [
"Mon, 23 Mar 2026 18:53:09 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["3006"],
"Server": ["istio-envoy"],
},
Body: <*exported.nopClosingBytesReader | 0xc0008c0b40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-rhqc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/nodePools/ephemeral-np",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-23T18:52:45Z\"}",
],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Authorization": ["redacted"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:167]: Unexpected error:
<*fmt.wrapError | 0xc000e04780>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-rl8trq" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000e04760>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00155d230>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"df7e757d-5e64-40a7-ade1-a33a21eb48eb",
],
"Date": [
"Mon, 23 Mar 2026 18:53:15 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2823"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc000e04600>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:167]: Unexpected error:
<*fmt.wrapError | 0xc000e04780>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-rl8trq" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create NodePool np-1: failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000e04760>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rl8trq\" to finish creating: GET https://rp.j7356928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rl8trq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00155d230>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"df7e757d-5e64-40a7-ade1-a33a21eb48eb",
],
"Date": [
"Mon, 23 Mar 2026 18:53:15 GMT",
],
"Content-Length": ["107"],
"X-Envoy-Upstream-Service-Time": ["2823"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc000e04600>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "GET",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j7356928.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (9)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 5 | 1.33%5 of 375 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 1 · Mar 28: 5 · Mar 29: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001060228>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-5vsdzh" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000c2afa0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c2af60>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c2af20>{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2b37d0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001060228>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-5vsdzh" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000c2afa0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c2af60>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000c2af20>{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2b37d0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013cc180>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-j9fds8" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000b0df20>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b0df00>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b0dee0>{
msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2b37d0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013cc180>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-j9fds8" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000b0df20>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b0df00>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000b0dee0>{
msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2b37d0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00017e150>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-8s5fww" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000969680>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000969660>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000969640>{
msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2b17d0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00017e150>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-8s5fww" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000969680>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000969660>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000969640>{
msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2b17d0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (5)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 4 | 1.07%4 of 375 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 2 · Mar 25: 1 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
<*fmt.wrapErrors | 0xc000d048d0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vsdn4b" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0002c7a30>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b",
},
<*fmt.wrapError | 0xc00145e300>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
<*fmt.wrapErrors | 0xc000d048d0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vsdn4b" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0002c7a30>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b",
},
<*fmt.wrapError | 0xc00145e300>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:145]: Unexpected error:
<*fmt.wrapErrors | 0xc000e2a300>:
failed to create NodePool np-d7hsqm, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-d7hsqm in resource group rg-stable-4-21-8bnd9n, error: failed waiting for nodepool="np-d7hsqm" for cluster "cluster-stable-4-21-d7hsqm" in resourcegroup="rg-stable-4-21-8bnd9n" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-d7hsqm, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-d7hsqm in resource group rg-stable-4-21-8bnd9n, error: failed waiting for nodepool=\"np-d7hsqm\" for cluster \"cluster-stable-4-21-d7hsqm\" in resourcegroup=\"rg-stable-4-21-8bnd9n\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b888e0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-d7hsqm in resource group rg-stable-4-21-8bnd9n",
},
<*fmt.wrapError | 0xc00013e220>{
msg: "failed waiting for nodepool=\"np-d7hsqm\" for cluster \"cluster-stable-4-21-d7hsqm\" in resourcegroup=\"rg-stable-4-21-8bnd9n\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:145]: Unexpected error:
<*fmt.wrapErrors | 0xc000e2a300>:
failed to create NodePool np-d7hsqm, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-d7hsqm in resource group rg-stable-4-21-8bnd9n, error: failed waiting for nodepool="np-d7hsqm" for cluster "cluster-stable-4-21-d7hsqm" in resourcegroup="rg-stable-4-21-8bnd9n" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-d7hsqm, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-d7hsqm in resource group rg-stable-4-21-8bnd9n, error: failed waiting for nodepool=\"np-d7hsqm\" for cluster \"cluster-stable-4-21-d7hsqm\" in resourcegroup=\"rg-stable-4-21-8bnd9n\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b888e0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-d7hsqm in resource group rg-stable-4-21-8bnd9n",
},
<*fmt.wrapError | 0xc00013e220>{
msg: "failed waiting for nodepool=\"np-d7hsqm\" for cluster \"cluster-stable-4-21-d7hsqm\" in resourcegroup=\"rg-stable-4-21-8bnd9n\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:143]: Unexpected error:
<*fmt.wrapErrors | 0xc00141aea0>:
failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-vl8jff" in resourcegroup="rg-np-version-upgrade-vl8jff-6ld5tp" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0005c2e60>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp",
},
<*fmt.wrapError | 0xc000a9b0c0>{
msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:143]: Unexpected error:
<*fmt.wrapErrors | 0xc00141aea0>:
failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-vl8jff" in resourcegroup="rg-np-version-upgrade-vl8jff-6ld5tp" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0005c2e60>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp",
},
<*fmt.wrapError | 0xc000a9b0c0>{
msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredContributing tests (3)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
E2E Jobs
42
Success Rate
47.62%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.47.62% (20/42)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 50.00% | 26 | integration/parallel | Customer should be able to use workload identity via the cluster OIDC issuer URL | |
| 59.46% | 37 | integration/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 62.50% | 40 | integration/parallel | Customer should be able to create an HCP cluster with back-level version 4.19 | |
| 62.50% | 40 | integration/parallel | Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle | |
| 65.00% | 40 | integration/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | e2e | 12 | 28.57%12 of 42 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsMar 22: 1 · Mar 23: 0 · Mar 24: 19 · Mar 25: 33 · Mar 26: 80 · Mar 27: 57 · Mar 28: 0 · Mar 29: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc0014cacc0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-ck2bxp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-ck2bxp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000929890>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc0014cacc0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-ck2bxp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-ck2bxp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000929890>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc0012dc8d0>:
failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster="ea-listk99r75" in resourcegroup="ea-list-7tsrmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001212330>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
},
<*fmt.wrapErrors | 0xc0012dc8a0>{
msg: "failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001212330>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc0012dc8d0>:
failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster="ea-listk99r75" in resourcegroup="ea-list-7tsrmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001212330>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
},
<*fmt.wrapErrors | 0xc0012dc8a0>{
msg: "failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001212330>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapErrors | 0xc0009fc450>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jnxjrh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000c71e70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
},
<*fmt.wrapErrors | 0xc0009fc420>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000c71e70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapErrors | 0xc0009fc450>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jnxjrh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000c71e70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
},
<*fmt.wrapErrors | 0xc0009fc420>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000c71e70>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (26)
Affected runs (12)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Interrupted by Userfull failure pattern: Interrupted by User Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | e2e | 11 | 26.19%11 of 42 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 4 · Mar 25: 12 · Mar 26: 23 · Mar 27: 17 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:127]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:127]: Interrupted by User Contributing tests (13)
Affected runs (11)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within 45 minutesfull failure pattern: Cluster should become ready within 45 minutes Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 10 | 23.81%10 of 42 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 1 · Mar 25: 2 · Mar 26: 4 · Mar 27: 3 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.002s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.002s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.003s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.003s.
Cluster should become ready within 45 minutes
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (10)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 8 | 19.05%8 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsMar 22: 0 · Mar 23: 1 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 5 · Mar 28: 15 · Mar 29: 24 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc001508390>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nqc9bn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-nqc9bn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001392170>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc001508390>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nqc9bn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-nqc9bn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001392170>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:156]: Unexpected error:
<*fmt.wrapErrors | 0xc00129fb00>:
failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2nlfhj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-2nlfhj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0007a6b60>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:156]: Unexpected error:
<*fmt.wrapErrors | 0xc00129fb00>:
failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2nlfhj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-2nlfhj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0007a6b60>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
<*fmt.wrapErrors | 0xc001211320>:
failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2m22rw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2m22rw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00092c660>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
<*fmt.wrapErrors | 0xc001211320>:
failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2m22rw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2m22rw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00092c660>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (18)
Affected runs (8)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Deserializaion Error: no output from commandfull failure pattern: Deserializaion Error: no output from command Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 7 | 16.67%7 of 42 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 2 · Mar 26: 3 · Mar 27: 2 · Mar 28: 0 · Mar 29: 1 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)e/proc.go:466
runtime.semacquire1(0xc001686608, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000a595f8 sp=0xc000a59590 pc=0x2b3451d
sync.runtime_SemacquireWaitGroup(0xc000a30c20?, 0x40?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000a59630 sp=0xc000a595f8 pc=0x2b59bae
sync.(*WaitGroup).Wait(0xc001686600)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000a59658 sp=0xc000a59630 pc=0x2b7d0e5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50}, {{0xc000daf270, 0x1, 0x1}, 0x34630b8a000, {0x7995009, 0x8}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc000a59768 sp=0xc000a59658 pc=0x4fdc233
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc000a59888 sp=0xc000a59768 pc=0x4fdbac5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f9abcaaa958?, 0xc00160ca50?})
<autogenerated>:1 +0x33 fp=0xc000a598b0 sp=0xc000a59888 pc=0x4fea753
runtime.call16(0xc0011e87e0, 0xc000091030, 0x0, 0x0, 0x0, 0x10, 0xc000a59b08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000a598d0 sp=0xc000a598b0 pc=0x2b5f163
runtime.reflectcall(0x73e36c0?, 0xc00160ca50?, 0x2?, 0x79c5cd8?, 0x0?, 0x12?, 0x7018be0?)
<autogenerated>:1 +0x36 fp=0xc000a59910 sp=0xc000a598d0 pc=0x2b679f6
reflect.Value.call({0x699dce0?, 0xc000091030?, 0xc00000029f?}, {0x7989dfb, 0x4}, {0xc000e30f60, 0x1, 0x2ae7368?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000a59e70 sp=0xc000a59910 pc=0x2b9a9c6
reflect.Value.Call({0x699dce0?, 0xc000091030?, 0x0?}, {0xc000e30f60?, 0x326f4c8?, 0xc000e00150?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000a59ee8 sp=0xc000a59e70 pc=0x2b99ab9
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8400760?, 0xc00160ca50?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000a59f88 sp=0xc000a59ee8 pc=0x2fba290
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000a59fe0 sp=0xc000a59f88 pc=0x2fccd5b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000a59fe8 sp=0xc000a59fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 104
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 1180 gp=0xc00137efc0 m=nil [select]:
runtime.gopark(0xc000095ab8?, 0x2?, 0x80?, 0xfa?, 0xc000095aac?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000095938 sp=0xc000095918 pc=0x2b5848e
runtime.selectgo(0xc000095ab8, 0xc000095aa8, 0xc0016104e0?, 0x0, 0x33570dc?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000095a78 sp=0xc000095938 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83ea848, 0xc000e9f4f0}, 0x79a98af?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000095ae8 sp=0xc000095a78 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x842fc60, {0x83ea848, 0xc000e9f4f0}, 0xc00095ec50)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x732 fp=0xc000095de8 sp=0xc000095ae8 pc=0x3792ed2
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster({0x83ea848, 0xc000e9f4a0}, 0xc001997380, {0xc000d69100, 0x1a}, {0xc0017bae70, 0x18}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc000095ef0 sp=0xc000095de8 pc=0x4fbdf38
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc000095f78 sp=0xc000095ef0 pc=0x4fc0534
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc000095fe0 sp=0xc000095f78 pc=0x3874450
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1178
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95
rax 0xfffffffffffffffc
rbx 0x5
rcx 0x2ad7e2e
rdx 0x80
rdi 0x5
rsi 0x7ffe140e4e5c
rbp 0x7ffe140e4e10
rsp 0x7ffe140e4dc8
r8 0x0
r9 0x0
r10 0x432
r11 0x246
r12 0x7ffe140e4ea0
r13 0x1
r14 0xb57b700
r15 0x7
rip 0x2ad7e2e
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command
e/proc.go:466
runtime.semacquire1(0xc001686608, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000a595f8 sp=0xc000a59590 pc=0x2b3451d
sync.runtime_SemacquireWaitGroup(0xc000a30c20?, 0x40?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000a59630 sp=0xc000a595f8 pc=0x2b59bae
sync.(*WaitGroup).Wait(0xc001686600)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000a59658 sp=0xc000a59630 pc=0x2b7d0e5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50}, {{0xc000daf270, 0x1, 0x1}, 0x34630b8a000, {0x7995009, 0x8}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc000a59768 sp=0xc000a59658 pc=0x4fdc233
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc000a59888 sp=0xc000a59768 pc=0x4fdbac5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f9abcaaa958?, 0xc00160ca50?})
<autogenerated>:1 +0x33 fp=0xc000a598b0 sp=0xc000a59888 pc=0x4fea753
runtime.call16(0xc0011e87e0, 0xc000091030, 0x0, 0x0, 0x0, 0x10, 0xc000a59b08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000a598d0 sp=0xc000a598b0 pc=0x2b5f163
runtime.reflectcall(0x73e36c0?, 0xc00160ca50?, 0x2?, 0x79c5cd8?, 0x0?, 0x12?, 0x7018be0?)
<autogenerated>:1 +0x36 fp=0xc000a59910 sp=0xc000a598d0 pc=0x2b679f6
reflect.Value.call({0x699dce0?, 0xc000091030?, 0xc00000029f?}, {0x7989dfb, 0x4}, {0xc000e30f60, 0x1, 0x2ae7368?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000a59e70 sp=0xc000a59910 pc=0x2b9a9c6
reflect.Value.Call({0x699dce0?, 0xc000091030?, 0x0?}, {0xc000e30f60?, 0x326f4c8?, 0xc000e00150?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000a59ee8 sp=0xc000a59e70 pc=0x2b99ab9
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8400760?, 0xc00160ca50?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000a59f88 sp=0xc000a59ee8 pc=0x2fba290
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000a59fe0 sp=0xc000a59f88 pc=0x2fccd5b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000a59fe8 sp=0xc000a59fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 104
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 1180 gp=0xc00137efc0 m=nil [select]:
runtime.gopark(0xc000095ab8?, 0x2?, 0x80?, 0xfa?, 0xc000095aac?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000095938 sp=0xc000095918 pc=0x2b5848e
runtime.selectgo(0xc000095ab8, 0xc000095aa8, 0xc0016104e0?, 0x0, 0x33570dc?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000095a78 sp=0xc000095938 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83ea848, 0xc000e9f4f0}, 0x79a98af?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000095ae8 sp=0xc000095a78 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x842fc60, {0x83ea848, 0xc000e9f4f0}, 0xc00095ec50)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x732 fp=0xc000095de8 sp=0xc000095ae8 pc=0x3792ed2
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster({0x83ea848, 0xc000e9f4a0}, 0xc001997380, {0xc000d69100, 0x1a}, {0xc0017bae70, 0x18}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc000095ef0 sp=0xc000095de8 pc=0x4fbdf38
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc000095f78 sp=0xc000095ef0 pc=0x4fc0534
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc000095fe0 sp=0xc000095f78 pc=0x3874450
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1178
/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95
rax 0xfffffffffffffffc
rbx 0x5
rcx 0x2ad7e2e
rdx 0x80
rdi 0x5
rsi 0x7ffe140e4e5c
rbp 0x7ffe140e4e10
rsp 0x7ffe140e4dc8
r8 0x0
r9 0x0
r10 0x432
r11 0x246
r12 0x7ffe140e4ea0
r13 0x1
r14 0xb57b700
r15 0x7
rip 0x2ad7e2e
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from commandternal.extractBodyFunction.func2({0x83ff000?, 0xc000852900?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 110
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 154 gp=0xc0002d6e00 m=nil [IO wait]:
runtime.gopark(0xc00128de20?, 0x202dc100?, 0x50?, 0x27?, 0xb?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000392720 sp=0xc000392700 pc=0x2b5848e
runtime.netpollblock(0x2bc2038?, 0x2ae4066?, 0x0?)
/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000392758 sp=0xc000392720 pc=0x2b17f37
internal/poll.runtime_pollWait(0x7f8021c36e00, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000392778 sp=0xc000392758 pc=0x2b56d25
internal/poll.(*pollDesc).wait(0xc000b26480?, 0xc000f12000?, 0x0)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0003927a0 sp=0xc000392778 pc=0x2be4d67
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b26480, {0xc000f12000, 0x3500, 0x3500})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000392838 sp=0xc0003927a0 pc=0x2be6199
net.(*netFD).Read(0xc000b26480, {0xc000f12000?, 0xc000f122c2?, 0x5?})
/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000392880 sp=0xc000392838 pc=0x2da9785
net.(*conn).Read(0xc0000bb868, {0xc000f12000?, 0x7f802021c948?, 0x7f806881f5c0?})
/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0003928c8 sp=0xc000392880 pc=0x2dbb825
crypto/tls.(*atLeastReader).Read(0xc000b105a0, {0xc000f12000?, 0x3239?, 0x2b551c5?})
/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000392910 sp=0xc0003928c8 pc=0x2e272bb
bytes.(*Buffer).ReadFrom(0xc000533ea8, {0x834e6c0, 0xc000b105a0})
/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000392968 sp=0xc000392910 pc=0x2c22d18
crypto/tls.(*Conn).readFromUntil(0xc000533c08, {0x834f220, 0xc0000bb868}, 0xc0003929d0?)
/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0003929a0 sp=0xc000392968 pc=0x2e2749e
crypto/tls.(*Conn).readRecordOrCCS(0xc000533c08, 0x0)
/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000392c18 sp=0xc0003929a0 pc=0x2e2451b
crypto/tls.(*Conn).readRecord(...)
/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000533c08, {0xc001591000, 0x1000, 0x3267100?})
/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000392c80 sp=0xc000392c18 pc=0x2e2ae05
bufio.(*Reader).Read(0xc001455a40, {0xc0012bf004, 0x9, 0x32751ae?})
/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000392cb8 sp=0xc000392c80 pc=0x2cbe4d7
io.ReadAtLeast({0x834dce0, 0xc001455a40}, {0xc0012bf004, 0x9, 0x9}, 0x9)
/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000392d00 sp=0xc000392cb8 pc=0x2bb2dce
io.ReadFull(...)
/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0012bf004, 0x9, 0xc00000018b?}, {0x834dce0?, 0xc001455a40?})
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000392d50 sp=0xc000392d00 pc=0x325dec5
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0012befc0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000392dd8 sp=0xc000392d50 pc=0x325ea8b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0012befc0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000392e18 sp=0xc000392dd8 pc=0x325f1f8
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000392fa8)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000392f60 sp=0xc000392e18 pc=0x32720ea
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001049dc0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000392fc8 sp=0xc000392f60 pc=0x3271472
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000392fe0 sp=0xc000392fc8 pc=0x326ac05
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000392fe8 sp=0xc000392fe0 pc=0x2b60c01
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 153
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5
rax 0xca
rbx 0x0
rcx 0x2b62a23
rdx 0x0
rdi 0xb57b7d0
rsi 0x80
rbp 0x7ffef1febfe0
rsp 0x7ffef1febf98
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x0
r13 0x1
r14 0xb5796e0
r15 0x3
rip 0x2b62a21
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command
ternal.extractBodyFunction.func2({0x83ff000?, 0xc000852900?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 110
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 154 gp=0xc0002d6e00 m=nil [IO wait]:
runtime.gopark(0xc00128de20?, 0x202dc100?, 0x50?, 0x27?, 0xb?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000392720 sp=0xc000392700 pc=0x2b5848e
runtime.netpollblock(0x2bc2038?, 0x2ae4066?, 0x0?)
/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000392758 sp=0xc000392720 pc=0x2b17f37
internal/poll.runtime_pollWait(0x7f8021c36e00, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000392778 sp=0xc000392758 pc=0x2b56d25
internal/poll.(*pollDesc).wait(0xc000b26480?, 0xc000f12000?, 0x0)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0003927a0 sp=0xc000392778 pc=0x2be4d67
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b26480, {0xc000f12000, 0x3500, 0x3500})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000392838 sp=0xc0003927a0 pc=0x2be6199
net.(*netFD).Read(0xc000b26480, {0xc000f12000?, 0xc000f122c2?, 0x5?})
/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000392880 sp=0xc000392838 pc=0x2da9785
net.(*conn).Read(0xc0000bb868, {0xc000f12000?, 0x7f802021c948?, 0x7f806881f5c0?})
/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0003928c8 sp=0xc000392880 pc=0x2dbb825
crypto/tls.(*atLeastReader).Read(0xc000b105a0, {0xc000f12000?, 0x3239?, 0x2b551c5?})
/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000392910 sp=0xc0003928c8 pc=0x2e272bb
bytes.(*Buffer).ReadFrom(0xc000533ea8, {0x834e6c0, 0xc000b105a0})
/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000392968 sp=0xc000392910 pc=0x2c22d18
crypto/tls.(*Conn).readFromUntil(0xc000533c08, {0x834f220, 0xc0000bb868}, 0xc0003929d0?)
/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0003929a0 sp=0xc000392968 pc=0x2e2749e
crypto/tls.(*Conn).readRecordOrCCS(0xc000533c08, 0x0)
/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000392c18 sp=0xc0003929a0 pc=0x2e2451b
crypto/tls.(*Conn).readRecord(...)
/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000533c08, {0xc001591000, 0x1000, 0x3267100?})
/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000392c80 sp=0xc000392c18 pc=0x2e2ae05
bufio.(*Reader).Read(0xc001455a40, {0xc0012bf004, 0x9, 0x32751ae?})
/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000392cb8 sp=0xc000392c80 pc=0x2cbe4d7
io.ReadAtLeast({0x834dce0, 0xc001455a40}, {0xc0012bf004, 0x9, 0x9}, 0x9)
/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000392d00 sp=0xc000392cb8 pc=0x2bb2dce
io.ReadFull(...)
/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0012bf004, 0x9, 0xc00000018b?}, {0x834dce0?, 0xc001455a40?})
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000392d50 sp=0xc000392d00 pc=0x325dec5
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0012befc0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000392dd8 sp=0xc000392d50 pc=0x325ea8b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0012befc0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000392e18 sp=0xc000392dd8 pc=0x325f1f8
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000392fa8)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000392f60 sp=0xc000392e18 pc=0x32720ea
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001049dc0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000392fc8 sp=0xc000392f60 pc=0x3271472
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000392fe0 sp=0xc000392fc8 pc=0x326ac05
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000392fe8 sp=0xc000392fe0 pc=0x2b60c01
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 153
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5
rax 0xca
rbx 0x0
rcx 0x2b62a23
rdx 0x0
rdi 0xb57b7d0
rsi 0x80
rbp 0x7ffef1febfe0
rsp 0x7ffef1febf98
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x0
r13 0x1
r14 0xb5796e0
r15 0x3
rip 0x2b62a21
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349 +0x905 fp=0xc000d96960 sp=0xc000d95d58 pc=0x2fb2dc5
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000b1ea88, {0x0, 0x0}, {0xb5a6520, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000a93320, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496 +0xa9b fp=0xc000d96e10 sp=0xc000d96960 pc=0x2fc68bb
github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68 +0x3c5 fp=0xc000d96eb8 sp=0xc000d96e10 pc=0x2fcdc25
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2ae7a85?, 0xc000610000?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85 +0x2ba fp=0xc000d97b88 sp=0xc000d96eb8 pc=0x5394dda
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x83e90e8, 0xc000c146e0}, 0xc000cf0a80, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596 +0xee fp=0xc000d97c30 sp=0xc000d97b88 pc=0x538b14e
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237 +0x196 fp=0xc000d97fe0 sp=0xc000d97c30 pc=0x5386ed6
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d97fe8 sp=0xc000d97fe0 pc=0x2b60c01
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230 +0x1e5
goroutine 96 gp=0xc000d58c40 m=nil [select]:
runtime.gopark(0xc000094ec8?, 0x2?, 0xc8?, 0xaa?, 0xc000094ebc?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000094d48 sp=0xc000094d28 pc=0x2b5848e
runtime.selectgo(0xc000094ec8, 0xc000094eb8, 0xc001486940?, 0x0, 0x33570dc?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000094e88 sp=0xc000094d48 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83e90e8, 0xc000909ae0}, 0x79a871d?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000094ef8 sp=0xc000094e88 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x841cda0, {0x83e90e8, 0xc000909ae0}, 0x0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x7b2 fp=0xc000095240 sp=0xc000094ef8 pc=0x383e6f2
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x83e9158?, 0xc0005293b0?}, {{0x83fef28?, 0xc0005fe200?}, 0xc0013f8ca8?}, 0xc000095748, {0xc0012c2960, 0x1e}, {0x7a1d023, 0x23}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694 +0x405 fp=0xc0000954b8 sp=0xc000095240 pc=0x4fc3965
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x7988ba7, 0x4}, ...}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346 +0x497 fp=0xc000095818 sp=0xc0000954b8 pc=0x4fb7717
github.com/Azure/ARO-HCP/test/e2e.init.func24.2({0x7f0ed0192000, 0xc00086afc0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:82 +0x486 fp=0xc000095f58 sp=0xc000095818 pc=0x51854e6
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x83ff000?, 0xc00086afc0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
rax 0xfffffffffffffffc
rbx 0x0
rcx 0x2b62a23
rdx 0x0
rdi 0xb57d910
rsi 0x80
rbp 0x7f0ed2249cb8
rsp 0x7f0ed2249c70
r8 0x0
r9 0x0
r10 0x7f0ed2249ca8
r11 0x206
r12 0x7f0ed2249ca8
r13 0x0
r14 0xc000002c40
r15 0x0
rip 0x2b62a23
rflags 0x206
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command
.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349 +0x905 fp=0xc000d96960 sp=0xc000d95d58 pc=0x2fb2dc5
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000b1ea88, {0x0, 0x0}, {0xb5a6520, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000a93320, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496 +0xa9b fp=0xc000d96e10 sp=0xc000d96960 pc=0x2fc68bb
github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68 +0x3c5 fp=0xc000d96eb8 sp=0xc000d96e10 pc=0x2fcdc25
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2ae7a85?, 0xc000610000?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85 +0x2ba fp=0xc000d97b88 sp=0xc000d96eb8 pc=0x5394dda
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x83e90e8, 0xc000c146e0}, 0xc000cf0a80, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596 +0xee fp=0xc000d97c30 sp=0xc000d97b88 pc=0x538b14e
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237 +0x196 fp=0xc000d97fe0 sp=0xc000d97c30 pc=0x5386ed6
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d97fe8 sp=0xc000d97fe0 pc=0x2b60c01
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230 +0x1e5
goroutine 96 gp=0xc000d58c40 m=nil [select]:
runtime.gopark(0xc000094ec8?, 0x2?, 0xc8?, 0xaa?, 0xc000094ebc?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000094d48 sp=0xc000094d28 pc=0x2b5848e
runtime.selectgo(0xc000094ec8, 0xc000094eb8, 0xc001486940?, 0x0, 0x33570dc?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000094e88 sp=0xc000094d48 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83e90e8, 0xc000909ae0}, 0x79a871d?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000094ef8 sp=0xc000094e88 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x841cda0, {0x83e90e8, 0xc000909ae0}, 0x0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x7b2 fp=0xc000095240 sp=0xc000094ef8 pc=0x383e6f2
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x83e9158?, 0xc0005293b0?}, {{0x83fef28?, 0xc0005fe200?}, 0xc0013f8ca8?}, 0xc000095748, {0xc0012c2960, 0x1e}, {0x7a1d023, 0x23}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694 +0x405 fp=0xc0000954b8 sp=0xc000095240 pc=0x4fc3965
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x7988ba7, 0x4}, ...}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346 +0x497 fp=0xc000095818 sp=0xc0000954b8 pc=0x4fb7717
github.com/Azure/ARO-HCP/test/e2e.init.func24.2({0x7f0ed0192000, 0xc00086afc0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:82 +0x486 fp=0xc000095f58 sp=0xc000095818 pc=0x51854e6
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x83ff000?, 0xc00086afc0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
rax 0xfffffffffffffffc
rbx 0x0
rcx 0x2b62a23
rdx 0x0
rdi 0xb57d910
rsi 0x80
rbp 0x7f0ed2249cb8
rsp 0x7f0ed2249c70
r8 0x0
r9 0x0
r10 0x7f0ed2249ca8
r11 0x206
r12 0x7f0ed2249ca8
r13 0x0
r14 0xc000002c40
r15 0x0
rip 0x2b62a23
rflags 0x206
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from commandContributing tests (6)
Affected runs (7)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 7.14%3 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 5 · Mar 29: 2 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
<*fmt.wrapError | 0xc0009e28a0>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008ed830>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Envoy-Upstream-Service-Time": ["855"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260329T123708Z:c6998271-44f0-41ca-8da6-1f46b5fe6948",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 29A874DFCC374E0391D1296600C0E257 Ref B: CH1AA2020617027 Ref C: 2026-03-29T12:37:07Z",
],
"Date": [
"Sun, 29 Mar 2026 12:37:08 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Request-Id": [
"5bfee52a-0937-45f3-82af-d95d755a8729",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/6a20fe18-7658-4e7c-bdbc-1e74dd3ff699",
],
"X-Ms-Correlation-Request-Id": [
"c6998271-44f0-41ca-8da6-1f46b5fe6948",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Pragma": ["no-cache"],
"Content-Length": ["107"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
},
Body: <*exported.nopClosingBytesReader | 0xc0009e27e0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "manageme...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
<*fmt.wrapError | 0xc0009e28a0>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008ed830>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Envoy-Upstream-Service-Time": ["855"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260329T123708Z:c6998271-44f0-41ca-8da6-1f46b5fe6948",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 29A874DFCC374E0391D1296600C0E257 Ref B: CH1AA2020617027 Ref C: 2026-03-29T12:37:07Z",
],
"Date": [
"Sun, 29 Mar 2026 12:37:08 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Request-Id": [
"5bfee52a-0937-45f3-82af-d95d755a8729",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/6a20fe18-7658-4e7c-bdbc-1e74dd3ff699",
],
"X-Ms-Correlation-Request-Id": [
"c6998271-44f0-41ca-8da6-1f46b5fe6948",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Pragma": ["no-cache"],
"Content-Length": ["107"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
},
Body: <*exported.nopClosingBytesReader | 0xc0009e27e0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "manageme...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
<*fmt.wrapError | 0xc0009aac00>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0010d7dd0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["560"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"8d6cae0d-8889-4205-aa31-fe38545dd223",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/1101af73-6be3-42a3-a250-ff1e67521ad4",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Sun, 29 Mar 2026 12:37:09 GMT",
],
"Pragma": ["no-cache"],
"Content-Length": ["107"],
"Expires": ["-1"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260329T123709Z:f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 55288A5C640E4D0B9D2013D93A2A6472 Ref B: CHI261140510031 Ref C: 2026-03-29T12:37:08Z",
],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
],
"X-Content-Type-Options": ["nosniff"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc0009aaac0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
<*fmt.wrapError | 0xc0009aac00>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0010d7dd0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["560"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"8d6cae0d-8889-4205-aa31-fe38545dd223",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/1101af73-6be3-42a3-a250-ff1e67521ad4",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Sun, 29 Mar 2026 12:37:09 GMT",
],
"Pragma": ["no-cache"],
"Content-Length": ["107"],
"Expires": ["-1"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260329T123709Z:f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 55288A5C640E4D0B9D2013D93A2A6472 Ref B: CHI261140510031 Ref C: 2026-03-29T12:37:08Z",
],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
],
"X-Content-Type-Options": ["nosniff"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc0009aaac0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
<*fmt.wrapError | 0xc000886fe0>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0006bc720>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Msedge-Ref": [
"Ref A: 5C18A5E5865A447CAD257B2D17AC69BC Ref B: CHI261140511062 Ref C: 2026-03-28T05:01:46Z",
],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"b8001b45-128e-4419-b622-ebd59339ca7f",
],
"X-Envoy-Upstream-Service-Time": ["588"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260328T050147Z:2938cc51-e278-4bbf-8d54-92e75318a0ab",
],
"Content-Length": ["107"],
"X-Ms-Failure-Cause": ["service"],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Sat, 28 Mar 2026 05:01:46 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/e2afbe11-ecaf-4b3c-9fb7-45c6d078ee48",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"2938cc51-e278-4bbf-8d54-92e75318a0ab",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Pragma": ["no-cache"],
},
Body: <*exported.nopClosingBytesReader | 0xc000886e40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
<*fmt.wrapError | 0xc000886fe0>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0006bc720>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Type": ["application/json"],
"X-Msedge-Ref": [
"Ref A: 5C18A5E5865A447CAD257B2D17AC69BC Ref B: CHI261140511062 Ref C: 2026-03-28T05:01:46Z",
],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"b8001b45-128e-4419-b622-ebd59339ca7f",
],
"X-Envoy-Upstream-Service-Time": ["588"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260328T050147Z:2938cc51-e278-4bbf-8d54-92e75318a0ab",
],
"Content-Length": ["107"],
"X-Ms-Failure-Cause": ["service"],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Sat, 28 Mar 2026 05:01:46 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/e2afbe11-ecaf-4b3c-9fb7-45c6d078ee48",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"2938cc51-e278-4bbf-8d54-92e75318a0ab",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Pragma": ["no-cache"],
},
Body: <*exported.nopClosingBytesReader | 0xc000886e40>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (5)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 3 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:144]: Unexpected error:
<*fmt.wrapErrors | 0xc000ed0a50>:
failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-n2zmdv" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ae53f0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv",
},
<*fmt.wrapError | 0xc0008d4d00>{
msg: "failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:144]: Unexpected error:
<*fmt.wrapErrors | 0xc000ed0a50>:
failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-n2zmdv" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ae53f0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv",
},
<*fmt.wrapError | 0xc0008d4d00>{
msg: "failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
<*fmt.wrapErrors | 0xc000971080>:
failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jbg5qp" in resourcegroup="rg-np-version-upgrade-jbg5qp-5qgd94" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009509c0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94",
},
<*fmt.wrapError | 0xc0006cafc0>{
msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
<*fmt.wrapErrors | 0xc000971080>:
failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jbg5qp" in resourcegroup="rg-np-version-upgrade-jbg5qp-5qgd94" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009509c0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94",
},
<*fmt.wrapError | 0xc0006cafc0>{
msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
<*fmt.wrapErrors | 0xc00141ce40>:
failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-vx4kj8" in resourcegroup="rg-np-version-upgrade-vx4kj8-59lpv4" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015989a0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4",
},
<*fmt.wrapError | 0xc001094800>{
msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
<*fmt.wrapErrors | 0xc00141ce40>:
failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-vx4kj8" in resourcegroup="rg-np-version-upgrade-vx4kj8-59lpv4" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015989a0>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4",
},
<*fmt.wrapError | 0xc001094800>{
msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredContributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscri...full failure pattern: ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000805a20>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-55ctkl" in resourcegroup="customer-rg-97sgrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000805a00>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001129e30>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000805a20>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-55ctkl" in resourcegroup="customer-rg-97sgrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000805a00>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001129e30>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000899900>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-rc7l8v" in resourcegroup="rg-cluster-nsg-subnet-reuse-5rm9ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008998e0>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e559e0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 2...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000899900>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-rc7l8v" in resourcegroup="rg-cluster-nsg-subnet-reuse-5rm9ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008998e0>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e559e0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 2...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message SAL validation failed with status code: 403; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message SAL validation failed with status code: 403; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapError | 0xc0008a5220>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zvtsqs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0",
"name": "424c0c62-70a3-4351-ac27-2c33c7bb83b0",
"status": "Failed",
"startTime": "2026-03-26T20:30:39.925830532Z",
"endTime": "2026-03-26T20:30:48.906575258Z",
"error": {
"code": "InternalServerError",
"message": "SAL validation failed with status code: 403"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"SAL validation failed with status code: 403\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008a5200>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"SAL validation failed with status code: 403\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011583c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 5A0DC58691384AA18C1C9F410C4E5748 Ref B: CHI261140508054 Ref C: 2026-03-26T20:30:50Z",
],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Correlation-Request-Id": [
"9ebf9896-a696-4ba0-a35b-c86547212be5",
],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"d7646fbd-d48e-43c2-9674-7365e0a2a6fa",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadaeast/b22816e7-8010-49c2-b249-f2327a9fe3fb",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapError | 0xc0008a5220>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zvtsqs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0",
"name": "424c0c62-70a3-4351-ac27-2c33c7bb83b0",
"status": "Failed",
"startTime": "2026-03-26T20:30:39.925830532Z",
"endTime": "2026-03-26T20:30:48.906575258Z",
"error": {
"code": "InternalServerError",
"message": "SAL validation failed with status code: 403"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"SAL validation failed with status code: 403\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008a5200>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"SAL validation failed with status code: 403\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0011583c0>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 5A0DC58691384AA18C1C9F410C4E5748 Ref B: CHI261140508054 Ref C: 2026-03-26T20:30:50Z",
],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Correlation-Request-Id": [
"9ebf9896-a696-4ba0-a35b-c86547212be5",
],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"d7646fbd-d48e-43c2-9674-7365e0a2a6fa",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadaeast/b22816e7-8010-49c2-b249-f2327a9fe3fb",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Ms-Routing-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapError | 0xc000aafca0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-qjbk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
"name": "34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
"status": "Failed",
"startTime": "2026-03-26T20:30:28.750332994Z",
"endTime": "2026-03-26T20:30:48.509300921Z",
"error": {
"code": "InternalServerError",
"message": "SAL validation failed with status code: 403"
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-qjbk7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n \"name\": \"34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-26T20:30:28.750332994Z\",\n \"endTime\": \"2026-03-26T20:30:48.509300921Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"SAL validation failed with status code: 403\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e36660>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Length": ["481"],
"X-Envoy-Upstream-Service-Time": ["18"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/d2b1b180-95e6-469a-a3cb-7097078b6027",
],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Request-Id": [
"886185d4-7470-4791-ba1e-90e0db1c201d",
],
"X-Ms-Correlation-Request-Id": [
"74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260326T203051Z:74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Type": ["application/json"],
"X-Msedge-Ref": [
"Ref A: 72447DB8C47A4BA58F26EB6D90519D0E Ref B: CHI261140501042 Ref C: 2026-03-26T20:30:51Z",
],
"Date": [
"Thu, 26 Mar 2026 20:30:51 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000aafb80>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 117, 107, 115, 111, 117, 116, 104, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 51, 52, 99, 100, 51, 102, 49, 57, 45, 57, 55, 102, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapError | 0xc000aafca0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-qjbk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
"name": "34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
"status": "Failed",
"startTime": "2026-03-26T20:30:28.750332994Z",
"endTime": "2026-03-26T20:30:48.509300921Z",
"error": {
"code": "InternalServerError",
"message": "SAL validation failed with status code: 403"
}
}
--------------------------------------------------------------------------------
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-qjbk7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n \"name\": \"34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-26T20:30:28.750332994Z\",\n \"endTime\": \"2026-03-26T20:30:48.509300921Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"SAL validation failed with status code: 403\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e36660>{
ErrorCode: "InternalServerError",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Length": ["481"],
"X-Envoy-Upstream-Service-Time": ["18"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/d2b1b180-95e6-469a-a3cb-7097078b6027",
],
"X-Content-Type-Options": ["nosniff"],
"X-Ms-Request-Id": [
"886185d4-7470-4791-ba1e-90e0db1c201d",
],
"X-Ms-Correlation-Request-Id": [
"74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260326T203051Z:74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Type": ["application/json"],
"X-Msedge-Ref": [
"Ref A: 72447DB8C47A4BA58F26EB6D90519D0E Ref B: CHI261140501042 Ref C: 2026-03-26T20:30:51Z",
],
"Date": [
"Thu, 26 Mar 2026 20:30:51 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000aafb80>{
s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 117, 107, 115, 111, 117, 116, 104, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 51, 52, 99, 100, 51, 102, 49, 57, 45, 57, 55, 102, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Resource is already deleting; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Resource is already deleting; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc00141cd38>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Resource is already deleting",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0008bb5e0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008bb5c0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00178e6c0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"28ede0cd-2936-4f2f-8253-cb605ecc9170",
],
"X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T152452Z:5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["295"],
"Content-Type": ["application/json"],
"X-Ms-Correlation-Request-Id": [
"5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
],
"Cache-Control": ["no-cache"],
"X-Ms-Error-Code": ["Conflict"],
"X-Envoy-Upstream-Service-Time": ["1026"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/5c5c915c-65f2-46d6-a279-f918cf82b3b5",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Del...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc00141cd38>:
failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Resource is already deleting",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*fmt.wrapError | 0xc0008bb5e0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008bb5c0>{
msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Resource is already deleting\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00178e6c0>{
ErrorCode: "Conflict",
StatusCode: 409,
RawResponse: {
Status: "409 Conflict",
StatusCode: 409,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"28ede0cd-2936-4f2f-8253-cb605ecc9170",
],
"X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T152452Z:5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["295"],
"Content-Type": ["application/json"],
"X-Ms-Correlation-Request-Id": [
"5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
],
"Cache-Control": ["no-cache"],
"X-Ms-Error-Code": ["Conflict"],
"X-Envoy-Upstream-Service-Time": ["1026"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/5c5c915c-65f2-46d6-a279-f918cf82b3b5",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Del...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc000f66e40>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-ct8x6p" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-ct8x6p\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001606080>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
<*fmt.wrapErrors | 0xc000f66e40>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-ct8x6p" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-ct8x6p\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001606080>{
s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnqfull failure pattern: found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc000d9d878>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq
{
errs: [
<*errors.errorString | 0xc000718bc0>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc000d9d878>:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq
{
errs: [
<*errors.errorString | 0xc000718bc0>{
s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:full failure pattern: operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster: Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 1 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
<*errors.errorString | 0xc000a33960>:
operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:
&armredhatopenshifthcp.HcpOpenShiftCluster{
Location: &"uksouth",
Identity: &{Type: &"UserAssigned", UserAssignedIdentities: {"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, ...}},
Properties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{
... // 7 identical fields
NodeDrainTimeoutMinutes: nil,
Version: &{ChannelGroup: &"stable", ID: &"4.20"},
- Console: &armredhatopenshifthcp.ConsoleProfile{
- URL: &"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net",
- },
+ Console: nil,
ProvisioningState: &"Succeeded",
},
Tags: {"aro-hcp.experimental.cluster.size-override": &"Minimal"},
ID: &"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou"...,
... // 1 ignored and 2 identical fields
}
{
s: "operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:\n\u00a0\u00a0&armredhatopenshifthcp.HcpOpenShiftCluster{\n\u00a0\u00a0\tLocation: &\"uksouth\",\n\u00a0\u00a0\tIdentity: &{Type: &\"UserAssigned\", UserAssignedIdentities: {\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, ...}},\n\u00a0\u00a0\tProperties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{\n\u00a0\u00a0\t\t... // 7 identical fields\n\u00a0\u00a0\t\tNodeDrainTimeoutMinutes: nil,\n\u00a0\u00a0\t\tVersion: &{ChannelGroup: &\"stable\", ID: &\"4.20\"},\n-\u00a0\t\tConsole: &armredhatopenshifthcp.ConsoleProfile{\n-\u00a0\t\t\tURL: &\"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net\",\n-\u00a0\t\t},\n+\u00a0\t\tConsole: nil,\n\u00a0\u00a0\t\tProvisioningState: &\"Succeeded\",\n\u00a0\u00a0\t},\n\u00a0\u00a0\tTags: {\"aro-hcp.experimental.cluster.size-override\": &\"Minimal\"},\n\u00a0\u00a0\tID: &\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou\"...,\n\u00a0\u00a0\t... // 1 ignored and 2 identical fields\n\u00a0\u00a0}\n",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
<*errors.errorString | 0xc000a33960>:
operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:
&armredhatopenshifthcp.HcpOpenShiftCluster{
Location: &"uksouth",
Identity: &{Type: &"UserAssigned", UserAssignedIdentities: {"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, ...}},
Properties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{
... // 7 identical fields
NodeDrainTimeoutMinutes: nil,
Version: &{ChannelGroup: &"stable", ID: &"4.20"},
- Console: &armredhatopenshifthcp.ConsoleProfile{
- URL: &"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net",
- },
+ Console: nil,
ProvisioningState: &"Succeeded",
},
Tags: {"aro-hcp.experimental.cluster.size-override": &"Minimal"},
ID: &"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou"...,
... // 1 ignored and 2 identical fields
}
{
s: "operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:\n\u00a0\u00a0&armredhatopenshifthcp.HcpOpenShiftCluster{\n\u00a0\u00a0\tLocation: &\"uksouth\",\n\u00a0\u00a0\tIdentity: &{Type: &\"UserAssigned\", UserAssignedIdentities: {\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, ...}},\n\u00a0\u00a0\tProperties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{\n\u00a0\u00a0\t\t... // 7 identical fields\n\u00a0\u00a0\t\tNodeDrainTimeoutMinutes: nil,\n\u00a0\u00a0\t\tVersion: &{ChannelGroup: &\"stable\", ID: &\"4.20\"},\n-\u00a0\t\tConsole: &armredhatopenshifthcp.ConsoleProfile{\n-\u00a0\t\t\tURL: &\"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net\",\n-\u00a0\t\t},\n+\u00a0\t\tConsole: nil,\n\u00a0\u00a0\t\tProvisioningState: &\"Succeeded\",\n\u00a0\u00a0\t},\n\u00a0\u00a0\tTags: {\"aro-hcp.experimental.cluster.size-override\": &\"Minimal\"},\n\u00a0\u00a0\tID: &\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou\"...,\n\u00a0\u00a0\t... // 1 ignored and 2 identical fields\n\u00a0\u00a0}\n",
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
E2E Jobs
19
Success Rate
42.11%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.44.44% (8/18)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 83.33% | 18 | stage/parallel | Customer should be able to create an HCP cluster | |
| 83.33% | 18 | stage/parallel | Customer should be able to create an HCP cluster and custom node pool osDisk size | |
| 88.89% | 18 | stage/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 88.89% | 18 | stage/parallel | Customer should be able to create an HCP cluster with back-level version 4.19 | |
| 88.89% | 18 | stage/parallel | Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 4 | 21.05%4 of 19 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsMar 22: 0 · Mar 23: 19 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapErrors | 0xc0008d40c0>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6wdcvw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-6wdcvw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008fb6d0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw",
},
<*fmt.wrapErrors | 0xc0008d4090>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-6wdcvw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008fb6d0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapErrors | 0xc0008d40c0>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6wdcvw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-6wdcvw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008fb6d0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw",
},
<*fmt.wrapErrors | 0xc0008d4090>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-6wdcvw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008fb6d0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-6wdcvw",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc00140a210>:
failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jlczdc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: context deadline exceeded
{
msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-jlczdc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000689460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc",
},
<*fmt.wrapErrors | 0xc00140a1e0>{
msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-jlczdc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000689460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc00140a210>:
failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jlczdc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: context deadline exceeded
{
msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-jlczdc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000689460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc",
},
<*fmt.wrapErrors | 0xc00140a1e0>{
msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-jlczdc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000689460>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-jlczdc",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
<*fmt.wrapErrors | 0xc000f04420>:
failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-q2pjbc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: context deadline exceeded
{
msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-q2pjbc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a0f880>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc",
},
<*fmt.wrapErrors | 0xc000f043f0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-q2pjbc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a0f880>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
<*fmt.wrapErrors | 0xc000f04420>:
failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-q2pjbc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: context deadline exceeded
{
msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-q2pjbc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a0f880>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc",
},
<*fmt.wrapErrors | 0xc000f043f0>{
msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-q2pjbc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a0f880>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-q2pjbc",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (18)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
node pool upgrade verification failed: <node> (version 4full failure pattern: node pool upgrade verification failed: <node> (version 4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 15.79%3 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 1 · Mar 28: 0 · Mar 29: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.001s.
Expected success, but got an error:
<*errors.errorString | 0xc001374b50>:
node pool upgrade verification failed: o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6 (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdf391b97b311ecf8f0e5be97024014ad8b264d7785313081aa1290eaf0bc100","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12e6985890d3c549f7cc9881892793b27277f72e2739cb910aaf9f4dd7215df1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab32e70fb149ae7470c941ca73bcfe9eba540bd5058a3d697a3ddf96597d0132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fadd21095ba14bd0d033a74eacbd06f7e40a03b9acbaa846a85a5dc16d73ca00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aadc8e5d5141737eec06abcb21778ea74ceca2076ed31a2b1277f332d837d2f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9eb3bea4e623795326319ebb95624f9b2a0905aac10ced9e74260931aa33ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6afe7e59c25f2df8e44971d20bf90820ff5c66abac524e656b2f84121032126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7db22f56cf9d223b1437e8cba3aa9e04e8ae8b973e23e85133bf1d3d8d66f15e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd202c6a40d89b2766a77a53ab1d84feb452d9f5d01066e3b7318ed2a13f8d54"]},{"name":"o0z2t2h0q3x4m1i-npupgrade-4-20-mb4xl-7xrtk","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"o0z2t2h0q3x4m1i-npupgrade-4-20-mb4xl-hsfj4","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6 (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.001s.
Expected success, but got an error:
<*errors.errorString | 0xc001374b50>:
node pool upgrade verification failed: o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6 (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdf391b97b311ecf8f0e5be97024014ad8b264d7785313081aa1290eaf0bc100","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12e6985890d3c549f7cc9881892793b27277f72e2739cb910aaf9f4dd7215df1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab32e70fb149ae7470c941ca73bcfe9eba540bd5058a3d697a3ddf96597d0132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fadd21095ba14bd0d033a74eacbd06f7e40a03b9acbaa846a85a5dc16d73ca00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aadc8e5d5141737eec06abcb21778ea74ceca2076ed31a2b1277f332d837d2f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9eb3bea4e623795326319ebb95624f9b2a0905aac10ced9e74260931aa33ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6afe7e59c25f2df8e44971d20bf90820ff5c66abac524e656b2f84121032126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7db22f56cf9d223b1437e8cba3aa9e04e8ae8b973e23e85133bf1d3d8d66f15e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd202c6a40d89b2766a77a53ab1d84feb452d9f5d01066e3b7318ed2a13f8d54"]},{"name":"o0z2t2h0q3x4m1i-npupgrade-4-20-mb4xl-7xrtk","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"o0z2t2h0q3x4m1i-npupgrade-4-20-mb4xl-hsfj4","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6 (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"o0z2t2h0q3x4m1i-npupgrade-4-20-dm7j5-v64n6\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
<*errors.errorString | 0xc00106ad10>:
node pool upgrade verification failed: w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]},{"name":"w0f3z0s5t4h0b1p-npupgrade-4-20-q5m24-hjflt","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"w0f3z0s5t4h0b1p-npupgrade-4-20-q5m24-m5nzs","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
<*errors.errorString | 0xc00106ad10>:
node pool upgrade verification failed: w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]},{"name":"w0f3z0s5t4h0b1p-npupgrade-4-20-q5m24-hjflt","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"w0f3z0s5t4h0b1p-npupgrade-4-20-q5m24-m5nzs","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"w0f3z0s5t4h0b1p-npupgrade-4-20-j6xzl-fttmd\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
<*errors.errorString | 0xc0010b9480>:
node pool upgrade verification failed: m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7983b528aa73931fdaa817858ecd416dea0ec5a6917bebf5bd11b4eb83c752c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]},{"name":"m8t3h5l2e3m8z3q-npupgrade-4-20-v5vdw-jcfj7","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"m8t3h5l2e3m8z3q-npupgrade-4-20-v5vdw-qsg2r","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c92...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
<*errors.errorString | 0xc0010b9480>:
node pool upgrade verification failed: m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7983b528aa73931fdaa817858ecd416dea0ec5a6917bebf5bd11b4eb83c752c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]},{"name":"m8t3h5l2e3m8z3q-npupgrade-4-20-v5vdw-jcfj7","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"m8t3h5l2e3m8z3q-npupgrade-4-20-v5vdw-qsg2r","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"m8t3h5l2e3m8z3q-npupgrade-4-20-mc27q-jplxt\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575\",\"arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c92...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installa...full failure pattern: ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 10.53%2 of 19 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsMar 22: 0 · Mar 23: 6 · Mar 24: 0 · Mar 25: 2 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapError | 0xc000bec1c0>:
failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-6n2bks": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-6n2bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: ServiceUnavailable
--------------------------------------------------------------------------------
{
"error": {
"code": "ServiceUnavailable",
"message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster patch-tags-cluster: failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-6n2bks\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-6n2bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bec1a0>{
msg: "failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-6n2bks\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-6n2bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000df6300>{
ErrorCode: "ServiceUnavailable",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Retry-After": ["59"],
"X-Ms-Error-Code": ["ServiceUnavailable"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/be034bc4-d201-4cf4-b728-556fe0ade3f9",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Date": [
"Wed, 25 Mar 2026 19:20:24 GMT",
],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"3ccd5529-b470-48dc-a2fe-ced3f6fe5dd5",
],
"X-Ms-Failure-Cause": ["service"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 899AC6984BA846F6A65A8506E85AD96A Ref B: CHI261140511034 Ref C: 2026-03-25T19:20:19Z",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["257"],
"X-Envoy-Upstream-Service-Time": ["4648"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Correlation-Request-Id": [
"9e757052-2886-416e-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapError | 0xc000bec1c0>:
failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-6n2bks": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-6n2bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: ServiceUnavailable
--------------------------------------------------------------------------------
{
"error": {
"code": "ServiceUnavailable",
"message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster patch-tags-cluster: failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-6n2bks\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-6n2bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bec1a0>{
msg: "failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-6n2bks\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-6n2bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000df6300>{
ErrorCode: "ServiceUnavailable",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Retry-After": ["59"],
"X-Ms-Error-Code": ["ServiceUnavailable"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/be034bc4-d201-4cf4-b728-556fe0ade3f9",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Date": [
"Wed, 25 Mar 2026 19:20:24 GMT",
],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"3ccd5529-b470-48dc-a2fe-ced3f6fe5dd5",
],
"X-Ms-Failure-Cause": ["service"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 899AC6984BA846F6A65A8506E85AD96A Ref B: CHI261140511034 Ref C: 2026-03-25T19:20:19Z",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["257"],
"X-Envoy-Upstream-Service-Time": ["4648"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Correlation-Request-Id": [
"9e757052-2886-416e-...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapError | 0xc000ad4120>:
failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-2nvstv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2nvstv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: ServiceUnavailable
--------------------------------------------------------------------------------
{
"error": {
"code": "ServiceUnavailable",
"message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2nvstv\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2nvstv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000ad4100>{
msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2nvstv\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2nvstv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000fca120>{
ErrorCode: "ServiceUnavailable",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["7588"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Pragma": ["no-cache"],
"Content-Length": ["257"],
"X-Ms-Error-Code": ["ServiceUnavailable"],
"X-Ms-Request-Id": [
"e8865110-76fd-4100-ac8b-61387f083b25",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T192115Z:c972a143-3485-47a2-9316-0276f2fe9c42",
],
"Cache-Control": ["no-cache"],
"Retry-After": ["59"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 80D5D109F1BA4D5AAB8CB91CE56B9D9A Ref B: CHI261140509025 Ref C: 2026-03-25T19:21:07Z",
],
"Content-Type": ["application/json"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/44ce2c96-32a8-470a-b768-16f9ab9bcbff",
],
"X-Ms-Correlation-Request-Id": [
"c972a143-3485-4...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapError | 0xc000ad4120>:
failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-2nvstv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2nvstv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: ServiceUnavailable
--------------------------------------------------------------------------------
{
"error": {
"code": "ServiceUnavailable",
"message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2nvstv\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2nvstv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000ad4100>{
msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2nvstv\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2nvstv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000fca120>{
ErrorCode: "ServiceUnavailable",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["7588"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Pragma": ["no-cache"],
"Content-Length": ["257"],
"X-Ms-Error-Code": ["ServiceUnavailable"],
"X-Ms-Request-Id": [
"e8865110-76fd-4100-ac8b-61387f083b25",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T192115Z:c972a143-3485-47a2-9316-0276f2fe9c42",
],
"Cache-Control": ["no-cache"],
"Retry-After": ["59"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 80D5D109F1BA4D5AAB8CB91CE56B9D9A Ref B: CHI261140509025 Ref C: 2026-03-25T19:21:07Z",
],
"Content-Type": ["application/json"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/44ce2c96-32a8-470a-b768-16f9ab9bcbff",
],
"X-Ms-Correlation-Request-Id": [
"c972a143-3485-4...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:100]: Unexpected error:
<*fmt.wrapError | 0xc000da82c0>:
failed starting cluster creation "admin-cred-lifecycle-bxcfml" in resourcegroup="admin-credential-lifecycle-test-66hgxn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-66hgxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-bxcfml
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: ServiceUnavailable
--------------------------------------------------------------------------------
{
"error": {
"code": "ServiceUnavailable",
"message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
}
}
--------------------------------------------------------------------------------
{
msg: "failed starting cluster creation \"admin-cred-lifecycle-bxcfml\" in resourcegroup=\"admin-credential-lifecycle-test-66hgxn\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-66hgxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-bxcfml\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001272600>{
ErrorCode: "ServiceUnavailable",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260323T103735Z:1d0494a6-08c9-402c-8237-38a0cda51cd9",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Expires": ["-1"],
"X-Ms-Error-Code": ["ServiceUnavailable"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["257"],
"X-Ms-Request-Id": [
"79dc0027-c979-49e8-87de-1be7d9fc9631",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/295cf239-fd2d-4d5e-8025-8d88e44a35af",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"Retry-After": ["59"],
"X-Envoy-Upstream-Service-Time": ["3453"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Msedge-Ref": [
"Ref A: 5FE01EDDE6B94A6EBE47085B54B20605 Ref B: CHI261140508023 Ref C: 2026-03-23T10:37:31Z",
],
"Date": [
"Mon, 23 Mar 2026 10:37:35 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"1d0494a6-08c9-402c-8237-38a0cda51cd9",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000da8200>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 83, 101, 114, 118, 105, 99, 101, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 105, 110, 115, 116, 97, 108, 108, 97, 116, 105, 111, 110, 32, 104, 97, 115, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 99, 97, 112, 97, 99, 105, 116, 121, 46, 32, 87, 104, 105, 108, 101, 32, 119, 101, 32, 97, 114, 101, 32, 98, 114, 105, 110, 103, 105, 110,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:100]: Unexpected error:
<*fmt.wrapError | 0xc000da82c0>:
failed starting cluster creation "admin-cred-lifecycle-bxcfml" in resourcegroup="admin-credential-lifecycle-test-66hgxn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-66hgxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-bxcfml
--------------------------------------------------------------------------------
RESPONSE 503: 503 Service Unavailable
ERROR CODE: ServiceUnavailable
--------------------------------------------------------------------------------
{
"error": {
"code": "ServiceUnavailable",
"message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
}
}
--------------------------------------------------------------------------------
{
msg: "failed starting cluster creation \"admin-cred-lifecycle-bxcfml\" in resourcegroup=\"admin-credential-lifecycle-test-66hgxn\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-66hgxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-bxcfml\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ServiceUnavailable\",\n \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001272600>{
ErrorCode: "ServiceUnavailable",
StatusCode: 503,
RawResponse: {
Status: "503 Service Unavailable",
StatusCode: 503,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260323T103735Z:1d0494a6-08c9-402c-8237-38a0cda51cd9",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Expires": ["-1"],
"X-Ms-Error-Code": ["ServiceUnavailable"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["257"],
"X-Ms-Request-Id": [
"79dc0027-c979-49e8-87de-1be7d9fc9631",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/295cf239-fd2d-4d5e-8025-8d88e44a35af",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"Retry-After": ["59"],
"X-Envoy-Upstream-Service-Time": ["3453"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Msedge-Ref": [
"Ref A: 5FE01EDDE6B94A6EBE47085B54B20605 Ref B: CHI261140508023 Ref C: 2026-03-23T10:37:31Z",
],
"Date": [
"Mon, 23 Mar 2026 10:37:35 GMT",
],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
"1d0494a6-08c9-402c-8237-38a0cda51cd9",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000da8200>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 83, 101, 114, 118, 105, 99, 101, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 105, 110, 115, 116, 97, 108, 108, 97, 116, 105, 111, 110, 32, 104, 97, 115, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 99, 97, 112, 97, 99, 105, 116, 121, 46, 32, 87, 104, 105, 108, 101, 32, 119, 101, 32, 97, 114, 101, 32, 98, 114, 105, 110, 103, 105, 110,...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (8)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscri...full failure pattern: ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0009a7a00>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-hf2tdd" in resourcegroup="rg-cluster-nsg-subnet-reuse-vrctxd" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-hf2tdd/operationStatuses/08584270510220855273
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-hf2tdd\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vrctxd\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-hf2tdd/operationStatuses/08584270510220855273\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0009a79e0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-hf2tdd\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vrctxd\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-hf2tdd/operationStatuses/08584270510220855273\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000de7800>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0009a7a00>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-hf2tdd" in resourcegroup="rg-cluster-nsg-subnet-reuse-vrctxd" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-hf2tdd/operationStatuses/08584270510220855273
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-hf2tdd\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vrctxd\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-hf2tdd/operationStatuses/08584270510220855273\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0009a79e0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-hf2tdd\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vrctxd\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-hf2tdd/operationStatuses/08584270510220855273\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-vrctxd/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000de7800>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000aa0980>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-jxmhhv" in resourcegroup="customer-rg-9njmp6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-9njmp6/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-jxmhhv/operationStatuses/08584270507671869181
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-jxmhhv\" in resourcegroup=\"customer-rg-9njmp6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-9njmp6/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-jxmhhv/operationStatuses/08584270507671869181\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000aa0960>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-jxmhhv\" in resourcegroup=\"customer-rg-9njmp6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-9njmp6/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-jxmhhv/operationStatuses/08584270507671869181\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e09c20>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000aa0980>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-jxmhhv" in resourcegroup="customer-rg-9njmp6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-9njmp6/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-jxmhhv/operationStatuses/08584270507671869181
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-jxmhhv\" in resourcegroup=\"customer-rg-9njmp6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-9njmp6/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-jxmhhv/operationStatuses/08584270507671869181\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000aa0960>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-jxmhhv\" in resourcegroup=\"customer-rg-9njmp6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-9njmp6/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-jxmhhv/operationStatuses/08584270507671869181\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-9njmp6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e09c20>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 2 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
<*fmt.wrapErrors | 0xc0011bc060>:
failed waiting for hcpCluster="cluster-zstream-4-20-2wqgr7" in resourcegroup="rg-zstream-upgrade-4-20-rdj65f" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-2wqgr7 in resource group rg-zstream-upgrade-4-20-rdj65f, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-2wqgr7\" in resourcegroup=\"rg-zstream-upgrade-4-20-rdj65f\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-2wqgr7 in resource group rg-zstream-upgrade-4-20-rdj65f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001591280>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-2wqgr7 in resource group rg-zstream-upgrade-4-20-rdj65f",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
<*fmt.wrapErrors | 0xc0011bc060>:
failed waiting for hcpCluster="cluster-zstream-4-20-2wqgr7" in resourcegroup="rg-zstream-upgrade-4-20-rdj65f" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-2wqgr7 in resource group rg-zstream-upgrade-4-20-rdj65f, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-2wqgr7\" in resourcegroup=\"rg-zstream-upgrade-4-20-rdj65f\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-2wqgr7 in resource group rg-zstream-upgrade-4-20-rdj65f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001591280>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-2wqgr7 in resource group rg-zstream-upgrade-4-20-rdj65f",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
<*fmt.wrapErrors | 0xc000f4e660>:
failed waiting for hcpCluster="admin-cred-lifecycle-g5kdq6" in resourcegroup="admin-credential-lifecycle-test-rsjp6l" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-g5kdq6 in resource group admin-credential-lifecycle-test-rsjp6l, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-g5kdq6\" in resourcegroup=\"admin-credential-lifecycle-test-rsjp6l\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-g5kdq6 in resource group admin-credential-lifecycle-test-rsjp6l, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008933b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-g5kdq6 in resource group admin-credential-lifecycle-test-rsjp6l",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
<*fmt.wrapErrors | 0xc000f4e660>:
failed waiting for hcpCluster="admin-cred-lifecycle-g5kdq6" in resourcegroup="admin-credential-lifecycle-test-rsjp6l" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-g5kdq6 in resource group admin-credential-lifecycle-test-rsjp6l, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-g5kdq6\" in resourcegroup=\"admin-credential-lifecycle-test-rsjp6l\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-g5kdq6 in resource group admin-credential-lifecycle-test-rsjp6l, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008933b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-g5kdq6 in resource group admin-credential-lifecycle-test-rsjp6l",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' is disabled; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' is disabled; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc000135720>:
failed to create HCP cluster cluster-zstream-4-19-t2pzfk: failed starting cluster creation "cluster-zstream-4-19-t2pzfk" in resourcegroup="rg-zstream-upgrade-4-19-jxxwdx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jxxwdx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-t2pzfk
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' is disabled"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-t2pzfk: failed starting cluster creation \"cluster-zstream-4-19-t2pzfk\" in resourcegroup=\"rg-zstream-upgrade-4-19-jxxwdx\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jxxwdx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-t2pzfk\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' is disabled\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000135700>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-t2pzfk\" in resourcegroup=\"rg-zstream-upgrade-4-19-jxxwdx\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jxxwdx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-t2pzfk\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' is disabled\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001202ba0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Wed, 25 Mar 2026 02:11:45 GMT",
],
"Content-Length": ["136"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T021146Z:9c7634db-566c-41f7-8d72-2aa3ce322e48",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: 721EFA18A062440EBE1322746807DA87 Ref B: CHI261140510040 Ref C: 2026-03-25T02:11:45Z",
],
"Pragma": ["no-cache"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"36238b70-47a0-49d4-8a7b-4444dbcf03d9",
],
"X-Envoy-Upstream-Service-Time": ["94"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Content-Type": ["application/json"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/32b35b19-98a7-4612-8e9c-f53aba55c90f",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc000135720>:
failed to create HCP cluster cluster-zstream-4-19-t2pzfk: failed starting cluster creation "cluster-zstream-4-19-t2pzfk" in resourcegroup="rg-zstream-upgrade-4-19-jxxwdx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jxxwdx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-t2pzfk
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' is disabled"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-t2pzfk: failed starting cluster creation \"cluster-zstream-4-19-t2pzfk\" in resourcegroup=\"rg-zstream-upgrade-4-19-jxxwdx\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jxxwdx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-t2pzfk\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' is disabled\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000135700>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-t2pzfk\" in resourcegroup=\"rg-zstream-upgrade-4-19-jxxwdx\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jxxwdx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-t2pzfk\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' is disabled\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001202ba0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Date": [
"Wed, 25 Mar 2026 02:11:45 GMT",
],
"Content-Length": ["136"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T021146Z:9c7634db-566c-41f7-8d72-2aa3ce322e48",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: 721EFA18A062440EBE1322746807DA87 Ref B: CHI261140510040 Ref C: 2026-03-25T02:11:45Z",
],
"Pragma": ["no-cache"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Request-Id": [
"36238b70-47a0-49d4-8a7b-4444dbcf03d9",
],
"X-Envoy-Upstream-Service-Time": ["94"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Content-Type": ["application/json"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/32b35b19-98a7-4612-8e9c-f53aba55c90f",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"X-Ms-Correlation-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Timed out after 120.301s.full failure pattern: Timed out after 120.301s. Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 1 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.301s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:162 with:
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.301s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:162 with:
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
get logs -n openshift-ingress router-default-<id> (deployment router-default): Get "<url>full failure pattern: get logs -n openshift-ingress router-default-<id> (deployment router-default): Get "<url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 1 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:207]: Timed out after 600.006s.
router-default deployment logs should be fetchable
Expected success, but got an error:
<*fmt.wrapError | 0xc0009a45e0>:
get logs -n openshift-ingress router-default-54886f7dd4-7hdvt (deployment router-default): Get "https://10.0.0.7:10250/containerLogs/openshift-ingress/router-default-54886f7dd4-7hdvt/router": EOF
{
msg: "get logs -n openshift-ingress router-default-54886f7dd4-7hdvt (deployment router-default): Get \"https://10.0.0.7:10250/containerLogs/openshift-ingress/router-default-54886f7dd4-7hdvt/router\": EOF",
err: <*errors.StatusError | 0xc000dde320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Get \"https://10.0.0.7:10250/containerLogs/openshift-ingress/router-default-54886f7dd4-7hdvt/router\": EOF",
Reason: "",
Details: nil,
Code: 500,
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:207]: Timed out after 600.006s.
router-default deployment logs should be fetchable
Expected success, but got an error:
<*fmt.wrapError | 0xc0009a45e0>:
get logs -n openshift-ingress router-default-54886f7dd4-7hdvt (deployment router-default): Get "https://10.0.0.7:10250/containerLogs/openshift-ingress/router-default-54886f7dd4-7hdvt/router": EOF
{
msg: "get logs -n openshift-ingress router-default-54886f7dd4-7hdvt (deployment router-default): Get \"https://10.0.0.7:10250/containerLogs/openshift-ingress/router-default-54886f7dd4-7hdvt/router\": EOF",
err: <*errors.StatusError | 0xc000dde320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Get \"https://10.0.0.7:10250/containerLogs/openshift-ingress/router-default-54886f7dd4-7hdvt/router\": EOF",
Reason: "",
Details: nil,
Code: 500,
},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
node pool upgrade verification failed: <node> (release images unchanged: [quayfull failure pattern: node pool upgrade verification failed: <node> (release images unchanged: [quay Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 1 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.008s.
Expected success, but got an error:
<*errors.errorString | 0xc000aa3e60>:
node pool upgrade verification failed: s7o1k0m7s5u5i3l-npupgrade-4-20-l2wk5-b6985 (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f63b50c9da4f329818c06abf12a5ab9c0d95d1edc5bd069a07f22caa5ea6886 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f09bfbfb4c18805a3f04d299b24ca7bdd56fa51bf694b5120d64c8cc47d5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04622e16d899255ce682524f5d96e4e4de47618e4b79bba78187f17bd02ef75e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06fd4850499ff4e69dc795f882307deefe2c72e89df63a065d411d11bb2b2e28 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fa7dae5d07b71b2e551c6d8b428f01265e171eb2bf6e10b3d68764fd9055d94 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:687dda94c5fb2fb22e845be36f8101c31da5718b134a929b6dad7a5943240f91 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:221e48cefff45243b181abc4074adf95df97712990030b4f280b65452083bae7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9d7abdce3802ef7144e0a880e1563fa40718223d8c664e0d572d0484892e872 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d1e25b874b260043556c33ed62a26ebd56bef929e3539fa86db96501a798b14 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4063cfa7fd0f3680e0cd433e7ea2795e729b1fcaaeb6725fc093659d40cab647 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a18b2d1cb3265f2e742d2356fff02c4034af7a0add21444612d6c77029175b18 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fada881117627ed34c8001afc157a42f6cf0efe3e4dda2b73ff8d665c2f8a878 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a2c4fd4e8ac98bf6b6cfd128f0a9378cebed09e095e520120fe5261165bae7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a973533a2ecfa414c1f95c4d8ffdbe85f18969a740c19d3b8c4d2096f17a94bf quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e976d4a7016dc2a541098fce90353d3bb922943373c8b820e5c95893ef739f8 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53801854d34f514ec3c26882830666cec06ff9e0b5145c081dece712a1a049fe quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25bca9158f1f47bae66a12dff46efe7a7e5c9f16235a946b209bc41fd758f7fc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e113ec3685f8d055c89e11b0ac8acc806b4aed59f19ae0eb4e7d896d4276a4c9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c1c12ad472c45ae03529792013154098ce4780a0fde6dfcc8b15674518b64e3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8ff003f97cba952bb70d5986702eb0753f39db65f141eaf612e743ce02a6d0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f4b6b4ddff81c2b6e1bb43e0924197ef184132ccb105ef816065bfebe25d002 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77988ac48ae4310b454503a6ad9a68516302560df62b97a7a1a59d182cca72dd arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d912fa096260c78f93b1a3f5b76686304ff1efa98c1b40e6ce93dcae97c7ba arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1bd55517958d9320e04f417ebcdae6ebd6b3fcc9d4cd683c54686c7660205df quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dcfd681099af2e4e7cbd5f68e5b100894ffca9ada4520ef3d9325d99f2fdf91 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6b881c99cea669ab664f41460b5910bbf8953e57dfafbdc5ef6b44d8b45dc39 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11bfcd233d69a8afc8cc1134aacedfd8f6457c3eaeebc40f0060f8792cea228f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9edb1bba41b757bc30ccc0bd7d823f3574e21195a1a4075a31ae5bea6fcea516 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2adfbbc4205be3d161063234991aae3f34ab6e0557207d9cfa4a498711a922b4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afa4fc3ab60c1c50f44c59e40f07fd06b52ddc65d3b87fb4bc9eb3e2d9a83aab quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28ea8251a1d39205b2007faf64f7ef00f6d2931b7b15906caf96124166047952 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:627843712da9222498c4ba152b6c1d9b4d77d55ca7205e782b84219fe23621c8 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246]); nodes=[{"name":"s7o1k0m7s5u5i3l-npupgrade-4-20-87gfn-j87gl","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25bca9158f1f47bae66a12dff46efe7a7e5c9f16235a946b209bc41fd758f7fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e113ec3685f8d055c89e11b0ac8acc806b4aed59f19ae0eb4e7d896d4276a4c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c1c12ad472c45ae03529792013154098ce4780a0fde6dfcc8b15674518b64e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8ff003f97cba952bb70d5986702eb0753f39db65f141eaf612e743ce02a6d0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a9e0e177155bd56a66707294fa9429241d6c66f63284e029528523a09a802c6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff01b00b64a65dc45eac4e3a607e7bf70e41fe333044f357e436dfa0fc4abcd1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:265112268f914a12e152e58a674116408253b120a35d5e696c2afd059b6fc015","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3113cec71482e3fcfdcac18c4affc122cbc46d68801dce3a5d1a03524ee711a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dcfd681099af2e4e7cbd5f68e5b100894ffca9ada4520ef3d9325d99f2fdf91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6b881c99cea669ab664f41460b5910bbf8953e57dfafbdc5ef6b44d8b45dc39","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:446273488a1ce22fd50f15a02afd8f415450c070d77b6af0c7bbef18f9666670","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:884468e937461054e4d0f6fbe428c0837c547fdfb77494fc4fd0d80c85ebc116","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2adfbbc4205be3d161063234991aae3f34ab6e0557207d9cfa4a498711a922b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afa4fc3ab60c1c50f44c59e40f07fd06b52ddc65d3b87fb4bc9eb3e2d9a83aab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246"]},{"name":"s7o1k0m7s5u5i3l-npupgrade-4-20-87gfn-zgsrj","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fa7dae5d07b71b2e551c6d8b428f01265e171eb2bf6e10b3d68764fd9055d94","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:687dda94c5fb2fb22e845be36f8101c31da5718b134a929b6dad7a5943240f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:221e48cefff45243b181abc4074adf95df97712990030b4f280b65452083bae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9d7abdce3802ef7144e0a880e1563fa40718223d8c664e0d572d0484892e872","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d1e25b874b260043556c33ed62a26ebd56bef929e3539fa86db96501a798b14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4063cfa7fd0f3680e0cd433e7ea2795e729b1fcaaeb6725fc093659d40cab647","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a18b2d1cb3265f2e742d2356fff02c4034af7a0add21444612d6c77029175b18","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fada881117627ed34c8001afc157a42f6cf0efe3e4dda2b73ff8d665c2f8a878","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a2c4fd4e8ac98bf6b6cfd128f0a9378cebed09e095e520120fe5261165bae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a973533a2ecfa414c1f95c4d8ffdbe85f18969a740c19d3b8c4d2096f17a94bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e976d4a7016dc2a541098fce90353d3bb922943373c8b820e5c95893ef739f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53801854d34f514ec3c26882830666cec06ff9e0b5145c081dece712a1a049fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d912fa096260c78f93b1a3f5b76686304ff1efa98c1b40e6ce93dcae97c7ba","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1bd55517958d9320e04f417ebcdae6ebd6b3fcc9d4cd683c54686c7660205df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11bfcd233d69a8afc8cc1134aacedfd8f6457c3eaeebc40f0060f8792cea228f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9edb1bba41b757bc30ccc0bd7d823f3574e21195a1a4075a31ae5bea6fcea516","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246"]},{"name":"s7o1k0m7s5u5i3l-npupgrade-4-20-l2wk5-b6985","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-2.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f63b50c9da4f329818c06abf12a5ab9c0d95d1edc5bd069a07f22caa5ea6886","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f09bfbfb4c18805a3f04d299b24ca7bdd56fa51bf694b5120d64c8cc47d5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04622e16d899255ce682524f5d96e4e4de47618e4b79bba78187f17bd02ef75e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06fd4850499ff4e69dc795f882307deefe2c72e89df63a065d411d11bb2b2e28","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fa7dae5d07b71b2e551c6d8b428f01265e171eb2bf6e10b3d68764fd9055d94","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:687dda94c5fb2fb22e845be36f8101c31da5718b134a929b6dad7a5943240f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:221e48cefff45243b181abc4074adf95df97712990030b4f280b65452083bae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9d7abdce3802ef7144e0a880e1563fa40718223d8c664e0d572d0484892e872","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d1e25b874b260043556c33ed62a26ebd56bef929e3539fa86db96501a798b14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4063cfa7fd0f3680e0cd433e7ea2795e729b1fcaaeb6725fc093659d40cab647","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a18b2d1cb3265f2e742d2356fff02c4034af7a0add21444612d6c77029175b18","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fada881117627ed34c8001afc157a42f6cf0efe3e4dda2b73ff8d665c2f8a878","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a2c4fd4e8ac98bf6b6cfd128f0a9378cebed09e095e520120fe5261165bae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a973533a2ecfa414c1f95c4d8ffdbe85f18969a740c19d3b8c4d2096f17a94bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e976d4a7016dc2a541098fce90353d3bb922943373c8b820e5c95893ef739f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53801854d34f514ec3c26882830666cec06ff9e0b5145c081dece712a1a049fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25bca9158f1f47bae66a12dff46efe7a7e5c9f16235a946b209bc41fd758f7fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e113ec3685f8d055c89e11b0ac8acc806b4aed59f19ae0eb4e7d896d4276a4c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c1c12ad472c45ae03529792013154098ce4780a0fde6dfcc8b15674518b64e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8ff003f97cba952bb70d5986702eb0753f39db65f141eaf612e743ce02a6d0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f4b6b4ddff81c2b6e1bb43e0924197ef184132ccb105ef816065bfebe25d002","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77988ac48ae4310b454503a6ad9a68516302560df62b97a7a1a59d182cca72dd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d912fa096260c78f93b1a3f5b76686304ff1efa98c1b40e6ce93dcae97c7ba","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1bd55517958d9320e04f417ebcdae6ebd6b3fcc9d4cd683c54686c7660205df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dcfd681099af2e4e7cbd5f68e5b100894ffca9ada4520ef3d9325d99f2fdf91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6b881c99cea669ab664f41460b5910bbf8953e57dfafbdc5ef6b44d8b45dc39","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11bfcd233d69a8afc8cc1134aacedfd8f6457c3eaeebc40f0060f8792cea228f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9edb1bba41b757bc30ccc0bd7d823f3574e21195a1a4075a31ae5bea6fcea516","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2adfbbc4205be3d161063234991aae3f34ab6e0557207d9cfa4a498711a922b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afa4fc3ab60c1c50f44c59e40f07fd06b52ddc65d3b87fb4bc9eb3e2d9a83aab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28ea8251a1d39205b2007faf64f7ef00f6d2931b7b15906caf96124166047952","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:627843712da9222498c4ba152b6c1d9b4d77d55ca7205e782b84219fe23621c8","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246"]}]
{
s: "node pool upgrade verification failed: s7o1k0m7s5u5i3l-npupgrade-4-20-l2wk5-b6985 (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f63b50c9da4f329818c06abf12a5ab9c0d95d1edc5bd069a07f22caa5ea6886 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f09bfbfb4c18805a3f04d299b24ca7bdd56fa51bf694b5120d64c8cc47d5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04622e16d899255ce682524f5d96e4e4de47618e4b79bba78187f17bd02ef75e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06fd4850499ff4e69dc795f882307deefe2c72e89df63a065d411d11bb2b2e28 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08 qu...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.008s.
Expected success, but got an error:
<*errors.errorString | 0xc000aa3e60>:
node pool upgrade verification failed: s7o1k0m7s5u5i3l-npupgrade-4-20-l2wk5-b6985 (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f63b50c9da4f329818c06abf12a5ab9c0d95d1edc5bd069a07f22caa5ea6886 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f09bfbfb4c18805a3f04d299b24ca7bdd56fa51bf694b5120d64c8cc47d5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04622e16d899255ce682524f5d96e4e4de47618e4b79bba78187f17bd02ef75e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06fd4850499ff4e69dc795f882307deefe2c72e89df63a065d411d11bb2b2e28 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fa7dae5d07b71b2e551c6d8b428f01265e171eb2bf6e10b3d68764fd9055d94 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:687dda94c5fb2fb22e845be36f8101c31da5718b134a929b6dad7a5943240f91 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:221e48cefff45243b181abc4074adf95df97712990030b4f280b65452083bae7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9d7abdce3802ef7144e0a880e1563fa40718223d8c664e0d572d0484892e872 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d1e25b874b260043556c33ed62a26ebd56bef929e3539fa86db96501a798b14 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4063cfa7fd0f3680e0cd433e7ea2795e729b1fcaaeb6725fc093659d40cab647 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a18b2d1cb3265f2e742d2356fff02c4034af7a0add21444612d6c77029175b18 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fada881117627ed34c8001afc157a42f6cf0efe3e4dda2b73ff8d665c2f8a878 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a2c4fd4e8ac98bf6b6cfd128f0a9378cebed09e095e520120fe5261165bae7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a973533a2ecfa414c1f95c4d8ffdbe85f18969a740c19d3b8c4d2096f17a94bf quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e976d4a7016dc2a541098fce90353d3bb922943373c8b820e5c95893ef739f8 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53801854d34f514ec3c26882830666cec06ff9e0b5145c081dece712a1a049fe quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25bca9158f1f47bae66a12dff46efe7a7e5c9f16235a946b209bc41fd758f7fc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e113ec3685f8d055c89e11b0ac8acc806b4aed59f19ae0eb4e7d896d4276a4c9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c1c12ad472c45ae03529792013154098ce4780a0fde6dfcc8b15674518b64e3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8ff003f97cba952bb70d5986702eb0753f39db65f141eaf612e743ce02a6d0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f4b6b4ddff81c2b6e1bb43e0924197ef184132ccb105ef816065bfebe25d002 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77988ac48ae4310b454503a6ad9a68516302560df62b97a7a1a59d182cca72dd arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d912fa096260c78f93b1a3f5b76686304ff1efa98c1b40e6ce93dcae97c7ba arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1bd55517958d9320e04f417ebcdae6ebd6b3fcc9d4cd683c54686c7660205df quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dcfd681099af2e4e7cbd5f68e5b100894ffca9ada4520ef3d9325d99f2fdf91 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6b881c99cea669ab664f41460b5910bbf8953e57dfafbdc5ef6b44d8b45dc39 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11bfcd233d69a8afc8cc1134aacedfd8f6457c3eaeebc40f0060f8792cea228f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9edb1bba41b757bc30ccc0bd7d823f3574e21195a1a4075a31ae5bea6fcea516 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2adfbbc4205be3d161063234991aae3f34ab6e0557207d9cfa4a498711a922b4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afa4fc3ab60c1c50f44c59e40f07fd06b52ddc65d3b87fb4bc9eb3e2d9a83aab quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28ea8251a1d39205b2007faf64f7ef00f6d2931b7b15906caf96124166047952 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:627843712da9222498c4ba152b6c1d9b4d77d55ca7205e782b84219fe23621c8 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246]); nodes=[{"name":"s7o1k0m7s5u5i3l-npupgrade-4-20-87gfn-j87gl","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25bca9158f1f47bae66a12dff46efe7a7e5c9f16235a946b209bc41fd758f7fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e113ec3685f8d055c89e11b0ac8acc806b4aed59f19ae0eb4e7d896d4276a4c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c1c12ad472c45ae03529792013154098ce4780a0fde6dfcc8b15674518b64e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8ff003f97cba952bb70d5986702eb0753f39db65f141eaf612e743ce02a6d0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a9e0e177155bd56a66707294fa9429241d6c66f63284e029528523a09a802c6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff01b00b64a65dc45eac4e3a607e7bf70e41fe333044f357e436dfa0fc4abcd1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:265112268f914a12e152e58a674116408253b120a35d5e696c2afd059b6fc015","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3113cec71482e3fcfdcac18c4affc122cbc46d68801dce3a5d1a03524ee711a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dcfd681099af2e4e7cbd5f68e5b100894ffca9ada4520ef3d9325d99f2fdf91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6b881c99cea669ab664f41460b5910bbf8953e57dfafbdc5ef6b44d8b45dc39","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:446273488a1ce22fd50f15a02afd8f415450c070d77b6af0c7bbef18f9666670","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:884468e937461054e4d0f6fbe428c0837c547fdfb77494fc4fd0d80c85ebc116","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2adfbbc4205be3d161063234991aae3f34ab6e0557207d9cfa4a498711a922b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afa4fc3ab60c1c50f44c59e40f07fd06b52ddc65d3b87fb4bc9eb3e2d9a83aab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246"]},{"name":"s7o1k0m7s5u5i3l-npupgrade-4-20-87gfn-zgsrj","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fa7dae5d07b71b2e551c6d8b428f01265e171eb2bf6e10b3d68764fd9055d94","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:687dda94c5fb2fb22e845be36f8101c31da5718b134a929b6dad7a5943240f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:221e48cefff45243b181abc4074adf95df97712990030b4f280b65452083bae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9d7abdce3802ef7144e0a880e1563fa40718223d8c664e0d572d0484892e872","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d1e25b874b260043556c33ed62a26ebd56bef929e3539fa86db96501a798b14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4063cfa7fd0f3680e0cd433e7ea2795e729b1fcaaeb6725fc093659d40cab647","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a18b2d1cb3265f2e742d2356fff02c4034af7a0add21444612d6c77029175b18","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fada881117627ed34c8001afc157a42f6cf0efe3e4dda2b73ff8d665c2f8a878","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a2c4fd4e8ac98bf6b6cfd128f0a9378cebed09e095e520120fe5261165bae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a973533a2ecfa414c1f95c4d8ffdbe85f18969a740c19d3b8c4d2096f17a94bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e976d4a7016dc2a541098fce90353d3bb922943373c8b820e5c95893ef739f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53801854d34f514ec3c26882830666cec06ff9e0b5145c081dece712a1a049fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d912fa096260c78f93b1a3f5b76686304ff1efa98c1b40e6ce93dcae97c7ba","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1bd55517958d9320e04f417ebcdae6ebd6b3fcc9d4cd683c54686c7660205df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11bfcd233d69a8afc8cc1134aacedfd8f6457c3eaeebc40f0060f8792cea228f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9edb1bba41b757bc30ccc0bd7d823f3574e21195a1a4075a31ae5bea6fcea516","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246"]},{"name":"s7o1k0m7s5u5i3l-npupgrade-4-20-l2wk5-b6985","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-2.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f63b50c9da4f329818c06abf12a5ab9c0d95d1edc5bd069a07f22caa5ea6886","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f09bfbfb4c18805a3f04d299b24ca7bdd56fa51bf694b5120d64c8cc47d5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04622e16d899255ce682524f5d96e4e4de47618e4b79bba78187f17bd02ef75e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06fd4850499ff4e69dc795f882307deefe2c72e89df63a065d411d11bb2b2e28","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fa7dae5d07b71b2e551c6d8b428f01265e171eb2bf6e10b3d68764fd9055d94","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:687dda94c5fb2fb22e845be36f8101c31da5718b134a929b6dad7a5943240f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:221e48cefff45243b181abc4074adf95df97712990030b4f280b65452083bae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9d7abdce3802ef7144e0a880e1563fa40718223d8c664e0d572d0484892e872","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d1e25b874b260043556c33ed62a26ebd56bef929e3539fa86db96501a798b14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4063cfa7fd0f3680e0cd433e7ea2795e729b1fcaaeb6725fc093659d40cab647","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a18b2d1cb3265f2e742d2356fff02c4034af7a0add21444612d6c77029175b18","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fada881117627ed34c8001afc157a42f6cf0efe3e4dda2b73ff8d665c2f8a878","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e331f1ab9d2a592d160997296a1fae7e343b154646e594051d9757768025f0a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af05801e4082f887de2d89a685c2d13678d3ea7aefe1272ecad2085ef3d25b73","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3fac63d925327ae02db3c5afb3e0a98ea12feb309e113abed5db353faf4ba7c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb0a82e107689001d3d798f835bd111c65ea2f220e052b367401ec3354f137ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182402ad5468234dca690272044b015d0545229ebd84e85834f57c767eb96d84","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad2c301140b27c0fbed130b7a2b47ecdc2a15c22ac4aaeb961d64521a9623e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a2c4fd4e8ac98bf6b6cfd128f0a9378cebed09e095e520120fe5261165bae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a973533a2ecfa414c1f95c4d8ffdbe85f18969a740c19d3b8c4d2096f17a94bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e976d4a7016dc2a541098fce90353d3bb922943373c8b820e5c95893ef739f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53801854d34f514ec3c26882830666cec06ff9e0b5145c081dece712a1a049fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5935b86fb6e92713f31e23166028e56f491ef14d756d8deead5c1455e73537","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd4b72aaaf892109abb29961b61fb95c77ec0a7b09158d747c84116646c0005b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e990663296804e663afd4729bccd85ffabc57a3e8ee474200955b72e5ca2573","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6accf480e8645c70f53adf062189259549144af3a4c3d736b28c561dd538f0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b4db8392b4fd5d49b177c707bed1048c6ccc7564f2dea39ebafe492807884df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2c7803cc35249f9c59b32e7ff691c32db4fb0a10a64267f2e0650f1ceef7847","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25bca9158f1f47bae66a12dff46efe7a7e5c9f16235a946b209bc41fd758f7fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e113ec3685f8d055c89e11b0ac8acc806b4aed59f19ae0eb4e7d896d4276a4c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17546c1840d65f2ee98ac183705e6956f4131d3ff32cde615728f9f25882b603","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fa6525db7f43e0dc02152928985547e4e2830fe46e65e6c627f8cf5aad99cd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3322914e1219da147d36aee3121d4d4b42bb619419535942aa9108f6918d8e3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d64191bc7e7c39b8b7b57ce216f872e5747295efa4be45a91de70731b639bfc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c1c12ad472c45ae03529792013154098ce4780a0fde6dfcc8b15674518b64e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8ff003f97cba952bb70d5986702eb0753f39db65f141eaf612e743ce02a6d0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f4b6b4ddff81c2b6e1bb43e0924197ef184132ccb105ef816065bfebe25d002","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77988ac48ae4310b454503a6ad9a68516302560df62b97a7a1a59d182cca72dd","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d912fa096260c78f93b1a3f5b76686304ff1efa98c1b40e6ce93dcae97c7ba","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1bd55517958d9320e04f417ebcdae6ebd6b3fcc9d4cd683c54686c7660205df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd9ca3b6c119ebeeb38e5529407f1889ce2c06c11f15bbdb38955caf2a3a44d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:99d2b1943fb0f198951ae9f5bf5a687891f4f5ec0954cbd1c2e2c7338848c18c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dcfd681099af2e4e7cbd5f68e5b100894ffca9ada4520ef3d9325d99f2fdf91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6b881c99cea669ab664f41460b5910bbf8953e57dfafbdc5ef6b44d8b45dc39","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3dacfcffb285f7279149c80cc05cad33dd968305420330dfc00ab732757095e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f117e6c476d1e70d006f3a32f70c6e8dde636d40ecf6b350348315bc30feaa9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11bfcd233d69a8afc8cc1134aacedfd8f6457c3eaeebc40f0060f8792cea228f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9edb1bba41b757bc30ccc0bd7d823f3574e21195a1a4075a31ae5bea6fcea516","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25daab414833b6f1f0f91b7a55935ed5cfd7aaad5d2f0eb8674a06a603187059","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a3fb7ffe43e04a2698de16a22589da10ae0a1f3e140ab690f0a6a0485b415c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b212e792d1598a476afc0853e57d79f411a562c54b310dc3b1f0a0f60fd30d7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4f48095d3dbbf481a4f0044b496ade9fa59cdcd58f6aa9cbea97b3a77cfbb1d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:766ba5aefee0ade548842c06f8061fe8b849b76d16280e3c41225f21d2d0b6be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd98db1975607ae0b375a1970472adafb289e25af6128d87126d87f3ab09a4de","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c158d5aed8eacb567f91a57aa8dcefcac9d3317a278a1e7f6bd5952f82509829","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4754971cc6f907064a799f4845fc9f949a3068a5b818b23eb292c8346f9f2f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10e051b10f8001bc4344d39d32a8d0c46dddb798f6ca025a6de0e78ba5b70b96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e6f5b7aa7eeb9a04f01301f8d878cd9285f864f58c11818da73b156c61162dd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2adfbbc4205be3d161063234991aae3f34ab6e0557207d9cfa4a498711a922b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afa4fc3ab60c1c50f44c59e40f07fd06b52ddc65d3b87fb4bc9eb3e2d9a83aab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38bd513387b83d3388d063e797d347932a335306b9f65eb73b7fbfb478b95748","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f470b689dd424c530740e39763240c647e66408089faf2d1ab45dfe689ebaddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6014a9a744d98499f4e68c99a612229b148db39c40dfc0fc72a0b0fdbbe26626","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c465826603124ccd81eb26a19e9e0d8f2c0c6f55892af64d078c573a9a386224","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2616e7262674a5b052e6e076ec6fd7425f221d623d9a3a18e37bd60440955388","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5855659dd086fe497b927919984f9ebbf9e5ab0a110ecc3d761efc6d44846245","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28ea8251a1d39205b2007faf64f7ef00f6d2931b7b15906caf96124166047952","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:627843712da9222498c4ba152b6c1d9b4d77d55ca7205e782b84219fe23621c8","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246"]}]
{
s: "node pool upgrade verification failed: s7o1k0m7s5u5i3l-npupgrade-4-20-l2wk5-b6985 (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab33f9ad593ee146550951038b289973cb1b80e15fa146ab20e823c89f64616 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88ada79212c91ed056d0ebd5103d2789d2e1d1aa0cf226f151b72fda8ed31cde quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:209c650a9554320e454d74132ee60da9e27f1e00c6db9ee25b1c4fc611ada105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:666aaeff92117bf500aa2fd3cb6326d39e83da6f6d9d08b883aa0ff66d56fed5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50bcea372ffc760129133a8ee4897569cbfe64c725d9c86a735d3eb04daf1c58 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5dc458e034b48610c05a65784b4ce8731709969a25867c1174beb7bef0d43e7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f63b50c9da4f329818c06abf12a5ab9c0d95d1edc5bd069a07f22caa5ea6886 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f09bfbfb4c18805a3f04d299b24ca7bdd56fa51bf694b5120d64c8cc47d5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86dfa215430ea4c2286a8e9e0921f3a4ca14fac70ae6f5ad4f66636fd5620e0f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f87a551096cbae321a440b01254f1a14e50cb7237e2abea79245b19dc4cb41cc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04622e16d899255ce682524f5d96e4e4de47618e4b79bba78187f17bd02ef75e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06fd4850499ff4e69dc795f882307deefe2c72e89df63a065d411d11bb2b2e28 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a123826e43a4807205a59beb9d9f68749abc08e7d913e7163059baaa637ca4bd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb139bd20b7cd79ee16cf8968dfcb0a5a3398bcfa734dcc2c2ec47dda46a93c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df01b29e279c5fef750737c48041bfb4ef531599fa6ce556987ecdfbf8eb6767 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:263c194859f4e030d9e44ad5f810cfdff86b4a2a8f5c423b1d6ddfef1d5b7fbb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58a646461aaea1fc7522e11a0a4acc5ca784923581a2d15cd36319cc1d59d210 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84ed295a1fd729c6bfd8250116d752b4c3ad66c7bc01ca44408e7cb8a115b5e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856de54698e1af65d0a646b1211cc5136a7c01e9ece7e653cdeb60a72a49772c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:014817ed5dc24098d4933c1865bb7bced08d1d2372e11861c6fc578acc972628 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f64e57709acc5ebc392907106819ad3f8896c436d1afacd30c6952c071c97be quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aef7c437611ed1adcba12e9875f3fb2f094d5116d5a0449066b48f25131faea4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdd372695e131ef18136d80cc294c502c4f240fa826e1f11ef2e2b492972682d quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:5bbbce615fb2103b900b9eadf79abae0c23b3ea14f8a5d46d3ff63879ded4058 quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-20@sha256:7354ad82237d1518a2bc0642f5a62b9b27962f22827b7c8d67a68a202adb7422 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43623b0924477ad0016ac986a0405f1127d56a42214d65a18f2e59ff8927bbb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d07f82ed2b467215c084754ef1fdb349536bbe0e3c83e3d48a2ab7a97b7f355 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc67f706b8c23d554bd34a126c4d2a4d36f1a95829887e7fd1679412ff1c558a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecebe8f91837ebe91c2f9c159b54f6fea43fd7d750929e2a5b08fd1aee688582 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13feaae13069cd9a08e285e2d077b96bc75cffeeda2e3b665443c022750d68da quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ab46d33d3b0a432aef2c0036e56c9cb9919ddfe74355a67689731d627d7d08 qu...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.26%1 of 19 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
<*fmt.wrapError | 0xc000637420>:
route host was never found: Get "https://agnhost-e2e-serving-app-nxlgf.apps.aro.e5l4y9o6k9c4o7e.kadt.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-nxlgf.apps.aro.e5l4y9o6k9c4o7e.kadt.uksouthstaging.aroapp-hcp.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc000ddae70>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-nxlgf.apps.aro.e5l4y9o6k9c4o7e.kadt.uksouthstaging.aroapp-hcp.io",
Err: <*tls.CertificateVerificationError | 0xc000ddae40>{
UnverifiedCertificates: [
{
Raw: [48, 130, 3, 252, 48, 130, 2, 228, 160, 3, 2, 1, 2, 2, 8, 99, 64, 47, 79, 215, 200, 164, 59, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 53, 48, 50, 49, 54, 50, 49, 90, 23, 13, 50, 55, 48, 51, 50, 53, 48, 50, 49, 54, 50, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 141, 241, 47, 239, 62, 253, 98, 49, 137, 235, 185, 74, 86, 196, 103, 131, 206, 39, 48, 77, 76, 111, 24, 181, 219, 27, 206, 133, 221, 141, 181, 51, 67, 196, 197, 174, 93, 201, 148, 41, 42, 132, 114, 223, 15, 125, 178, 202, 215, 86, 233, 146, 214, 18, 59, 80, 209, 246, 19, 50, 54, 174, 218, 52, 71, 91, 90, 16, 99, 86, 253, 104, 196, 99, 142, 18, 2, 145, 74, 63, 102, 52, 131, 5, 40, 104, 109, 75, 65, 36, 33, 177, 94, 124, 226, 88, 123, 56, 5, 150, 84, 114, 168, 154, 37, 189, 66, 67, 107, 87, 251, 138, 196, 86, 62, 167, 168, 233, 72, 67, 100, 247, 25, 242, 70, 238, 216, 155, 97, 53, 61, 177, 177, 60, 115, 157, 153, 13, 110, 247, 94, 46, 56, 117, 93, 169, 81, 173, 168, 69, 51, 112, 138, 117, 67, 173, 153, 224, 191, 36, 121, 49, 98, 133, 124, 68, 143, 233, 221, 159, 132, 202, 2, 74, 155, 255, 29, 200, 226, 105, 25, 50, 38, 70, 52, 145, 157, 52, 28, 239, 243, 87, 28, 131, 208, 237, 232, 212, 74, 145, 33, 46, 74, 56, 28, 103, 183, 89, 242, 58, 148, 7, 82, 14, 167, 228, 10, 113, 90, 28, 0, 35, 44, 149, 27, 84, 143, 232, 30, 115, 121, 187, 144, 44, 19, 16, 153, 241, 196, 130, 18, 149, 235, 50, 215, 66, 238, 243, 124, 77, 7, 206, 104, 178, 220, 95, 2, 3, 1, 0, 1, 163, 130, 1, 34, 48, 130, 1, 30, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 20, 223, 115, 140, 30, 53, 210, 171, 227, 91, 106, 67, 254, 185, 30, 184, 153, 109, 180, 244, 255, 215, 85, 97, 81, 23, 89, 176, 168, 171, 191, 149, 157, 6, 212, 41, 106, 38, 68, 14, 11, 92, 9, 78, 68, 95, 81, 20, 190, 16, 147, 200, 24, 89, 195, 6, 231, 37, 115, 113, 15, 238, 67, 242, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 74, 208, 108, 214, 165, 118, 38, 97, 81, 164, 223, 165, 38, 47, 30, 56, 252, 121, 209, 250, 225, 42, 169, 242, 147, 184, 173, 252, 82, 6, 243, 244, 116, 7, 44, 171, 1, 53, 6, 13, 50, 127, 103, 100, 219, 242, 231, 141, 249, 165, 235, 141, 17, 206, 109, 28, 119, 33, 32, 124, 204, 158, 77, 120, 48, 71, 6, 3, 85, 29, 17, 4, 64, 48, 62, 130, 60, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 101, 53, 108, 52, 121, 57, 111, 54, 107, 57, 99, 52, 111, 55, 101, 46, 107, 97, 100, 116, 46, 117, 107, 115, 111, 117, 116, 104, 115, 116, 97, 103, 105, 110, 103, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 105, 111, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 13, 1, 98, 31, 151, 215, 217, 129, 223, 198, 164, 43, 112, 111, 115, 255, 79, 166, 50, 156, 171, 37, 126, 122, 145, 48, 254, 197, 3, 135, 241, 241, 181, 186, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
<*fmt.wrapError | 0xc000637420>:
route host was never found: Get "https://agnhost-e2e-serving-app-nxlgf.apps.aro.e5l4y9o6k9c4o7e.kadt.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-nxlgf.apps.aro.e5l4y9o6k9c4o7e.kadt.uksouthstaging.aroapp-hcp.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc000ddae70>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-nxlgf.apps.aro.e5l4y9o6k9c4o7e.kadt.uksouthstaging.aroapp-hcp.io",
Err: <*tls.CertificateVerificationError | 0xc000ddae40>{
UnverifiedCertificates: [
{
Raw: [48, 130, 3, 252, 48, 130, 2, 228, 160, 3, 2, 1, 2, 2, 8, 99, 64, 47, 79, 215, 200, 164, 59, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 53, 48, 50, 49, 54, 50, 49, 90, 23, 13, 50, 55, 48, 51, 50, 53, 48, 50, 49, 54, 50, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 141, 241, 47, 239, 62, 253, 98, 49, 137, 235, 185, 74, 86, 196, 103, 131, 206, 39, 48, 77, 76, 111, 24, 181, 219, 27, 206, 133, 221, 141, 181, 51, 67, 196, 197, 174, 93, 201, 148, 41, 42, 132, 114, 223, 15, 125, 178, 202, 215, 86, 233, 146, 214, 18, 59, 80, 209, 246, 19, 50, 54, 174, 218, 52, 71, 91, 90, 16, 99, 86, 253, 104, 196, 99, 142, 18, 2, 145, 74, 63, 102, 52, 131, 5, 40, 104, 109, 75, 65, 36, 33, 177, 94, 124, 226, 88, 123, 56, 5, 150, 84, 114, 168, 154, 37, 189, 66, 67, 107, 87, 251, 138, 196, 86, 62, 167, 168, 233, 72, 67, 100, 247, 25, 242, 70, 238, 216, 155, 97, 53, 61, 177, 177, 60, 115, 157, 153, 13, 110, 247, 94, 46, 56, 117, 93, 169, 81, 173, 168, 69, 51, 112, 138, 117, 67, 173, 153, 224, 191, 36, 121, 49, 98, 133, 124, 68, 143, 233, 221, 159, 132, 202, 2, 74, 155, 255, 29, 200, 226, 105, 25, 50, 38, 70, 52, 145, 157, 52, 28, 239, 243, 87, 28, 131, 208, 237, 232, 212, 74, 145, 33, 46, 74, 56, 28, 103, 183, 89, 242, 58, 148, 7, 82, 14, 167, 228, 10, 113, 90, 28, 0, 35, 44, 149, 27, 84, 143, 232, 30, 115, 121, 187, 144, 44, 19, 16, 153, 241, 196, 130, 18, 149, 235, 50, 215, 66, 238, 243, 124, 77, 7, 206, 104, 178, 220, 95, 2, 3, 1, 0, 1, 163, 130, 1, 34, 48, 130, 1, 30, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 20, 223, 115, 140, 30, 53, 210, 171, 227, 91, 106, 67, 254, 185, 30, 184, 153, 109, 180, 244, 255, 215, 85, 97, 81, 23, 89, 176, 168, 171, 191, 149, 157, 6, 212, 41, 106, 38, 68, 14, 11, 92, 9, 78, 68, 95, 81, 20, 190, 16, 147, 200, 24, 89, 195, 6, 231, 37, 115, 113, 15, 238, 67, 242, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 74, 208, 108, 214, 165, 118, 38, 97, 81, 164, 223, 165, 38, 47, 30, 56, 252, 121, 209, 250, 225, 42, 169, 242, 147, 184, 173, 252, 82, 6, 243, 244, 116, 7, 44, 171, 1, 53, 6, 13, 50, 127, 103, 100, 219, 242, 231, 141, 249, 165, 235, 141, 17, 206, 109, 28, 119, 33, 32, 124, 204, 158, 77, 120, 48, 71, 6, 3, 85, 29, 17, 4, 64, 48, 62, 130, 60, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 101, 53, 108, 52, 121, 57, 111, 54, 107, 57, 99, 52, 111, 55, 101, 46, 107, 97, 100, 116, 46, 117, 107, 115, 111, 117, 116, 104, 115, 116, 97, 103, 105, 110, 103, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 105, 111, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 13, 1, 98, 31, 151, 215, 217, 129, 223, 198, 164, 43, 112, 111, 115, 255, 79, 166, 50, 156, 171, 37, 126, 122, 145, 48, 254, 197, 3, 135, 241, 241, 181, 186, ...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
E2E Jobs
43
Success Rate
53.49%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.56.10% (23/41)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 80.00% | 20 | prod/parallel | Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 6 | 13.95%6 of 43 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 5 · Mar 27: 1 · Mar 28: 1 · Mar 29: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error:
<*fmt.wrapErrors | 0xc0013db710>:
failed to create HCP cluster cluster-zstream-4-20-jgq7cv, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: failed waiting for cluster="cluster-zstream-4-20-jgq7cv" in resourcegroup="rg-zstream-upgrade-4-20-n5q24m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: context deadline exceeded
{
msg: "failed to create HCP cluster cluster-zstream-4-20-jgq7cv, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: failed waiting for cluster=\"cluster-zstream-4-20-jgq7cv\" in resourcegroup=\"rg-zstream-upgrade-4-20-n5q24m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0012e33b0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m",
},
<*fmt.wrapErrors | 0xc0013db6e0>{
msg: "failed waiting for cluster=\"cluster-zstream-4-20-jgq7cv\" in resourcegroup=\"rg-zstream-upgrade-4-20-n5q24m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0012e33b0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error:
<*fmt.wrapErrors | 0xc0013db710>:
failed to create HCP cluster cluster-zstream-4-20-jgq7cv, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: failed waiting for cluster="cluster-zstream-4-20-jgq7cv" in resourcegroup="rg-zstream-upgrade-4-20-n5q24m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: context deadline exceeded
{
msg: "failed to create HCP cluster cluster-zstream-4-20-jgq7cv, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: failed waiting for cluster=\"cluster-zstream-4-20-jgq7cv\" in resourcegroup=\"rg-zstream-upgrade-4-20-n5q24m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0012e33b0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m",
},
<*fmt.wrapErrors | 0xc0013db6e0>{
msg: "failed waiting for cluster=\"cluster-zstream-4-20-jgq7cv\" in resourcegroup=\"rg-zstream-upgrade-4-20-n5q24m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0012e33b0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-jgq7cv in resource group rg-zstream-upgrade-4-20-n5q24m",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc0011a00f0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6fc2f8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-6fc2f8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-6fc2f8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-6fc2f8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fdfac0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-6fc2f8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc0011a00f0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6fc2f8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-6fc2f8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-6fc2f8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-6fc2f8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fdfac0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-6fc2f8",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc000dbc090>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vg7dpc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-vg7dpc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011c6190>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc000dbc090>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vg7dpc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-vg7dpc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011c6190>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (5)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 4 | 9.30%4 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 2 · Mar 25: 2 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0004ec240>:
failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation "cluster-zstream-4-19-ttlq29" in resourcegroup="rg-zstream-upgrade-4-19-dmw26l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0004ec200>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00110c240>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: 2550DE417C1E449D8F0C03A8300EB8A9 Ref B: CHI261140502040 Ref C: 2026-03-25T08:38:47Z",
],
"Date": [
"Wed, 25 Mar 2026 08:38:48 GMT",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["138"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/b10cf018-156c-4dbb-acc4-7a2bd061c929",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"1ca5abf8-7530-42ab-b8ed-908d89ca29c9",
],
"X-Ms-Correlation-Request-Id": [
"bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
],
"X-Ms-Routing-Request-Id": [
"WESTEUROPE:20260325T083849Z:bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Type": ["ap...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0004ec240>:
failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation "cluster-zstream-4-19-ttlq29" in resourcegroup="rg-zstream-upgrade-4-19-dmw26l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0004ec200>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc00110c240>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Msedge-Ref": [
"Ref A: 2550DE417C1E449D8F0C03A8300EB8A9 Ref B: CHI261140502040 Ref C: 2026-03-25T08:38:47Z",
],
"Date": [
"Wed, 25 Mar 2026 08:38:48 GMT",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["138"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/b10cf018-156c-4dbb-acc4-7a2bd061c929",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"1ca5abf8-7530-42ab-b8ed-908d89ca29c9",
],
"X-Ms-Correlation-Request-Id": [
"bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
],
"X-Ms-Routing-Request-Id": [
"WESTEUROPE:20260325T083849Z:bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Type": ["ap...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0009b92a0>:
failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation "cluster-zstream-4-19-gs2t7t" in resourcegroup="rg-zstream-upgrade-4-19-gtmk6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0009b9280>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000ed84e0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"00a7a86a-0829-4052-a505-f4a3f7fbe820",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Envoy-Upstream-Service-Time": ["47"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Pragma": ["no-cache"],
"Content-Length": ["138"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T024457Z:205f05d7-f7ea-4800-9e3a-0982d1258690",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: B109C53224CE4681B497F6C095BB9092 Ref B: CHI261140510062 Ref C: 2026-03-25T02:44:56Z",
],
"Date": [
"Wed, 25 Mar 2026 02:44:56 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/b6b931c5-6267-442b-bbcc-c343b9c46261",
],
"X-Ms-Correlation-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0009b92a0>:
failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation "cluster-zstream-4-19-gs2t7t" in resourcegroup="rg-zstream-upgrade-4-19-gtmk6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0009b9280>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000ed84e0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"00a7a86a-0829-4052-a505-f4a3f7fbe820",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Envoy-Upstream-Service-Time": ["47"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Pragma": ["no-cache"],
"Content-Length": ["138"],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T024457Z:205f05d7-f7ea-4800-9e3a-0982d1258690",
],
"X-Content-Type-Options": ["nosniff"],
"X-Msedge-Ref": [
"Ref A: B109C53224CE4681B497F6C095BB9092 Ref B: CHI261140510062 Ref C: 2026-03-25T02:44:56Z",
],
"Date": [
"Wed, 25 Mar 2026 02:44:56 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/b6b931c5-6267-442b-bbcc-c343b9c46261",
],
"X-Ms-Correlation-Request-Id": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0009009c0>:
failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation "cluster-zstream-4-19-q94v6k" in resourcegroup="rg-zstream-upgrade-4-19-9pkbw7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000900980>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012389c0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"f19d9e44-6f27-4377-a922-81ce37e678a2",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260325T001907Z:f19d9e44-6f27-4377-a922-81ce37e678a2",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Length": ["138"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/58824f7e-b158-49e9-8f26-b1f9ce14eace",
],
"Date": [
"Wed, 25 Mar 2026 00:19:06 GMT",
],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"7087b8c2-6c1c-4159-98ea-9c1c04639dc3",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 86F1E95FE862488BABE9E655ECC14C19 Ref B: CHI261140501060 Ref C: 2026-03-25T00:19:06Z",
],
"Cache-Control": ["no-cache"],
"Content-Type"...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
<*fmt.wrapError | 0xc0009009c0>:
failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation "cluster-zstream-4-19-q94v6k" in resourcegroup="rg-zstream-upgrade-4-19-9pkbw7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000900980>{
msg: "failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0012389c0>{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"f19d9e44-6f27-4377-a922-81ce37e678a2",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260325T001907Z:f19d9e44-6f27-4377-a922-81ce37e678a2",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Content-Length": ["138"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/58824f7e-b158-49e9-8f26-b1f9ce14eace",
],
"Date": [
"Wed, 25 Mar 2026 00:19:06 GMT",
],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"7087b8c2-6c1c-4159-98ea-9c1c04639dc3",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Msedge-Ref": [
"Ref A: 86F1E95FE862488BABE9E655ECC14C19 Ref B: CHI261140501060 Ref C: 2026-03-25T00:19:06Z",
],
"Cache-Control": ["no-cache"],
"Content-Type"...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
Deserializaion Error: no output from commandfull failure pattern: Deserializaion Error: no output from command Signal: NoiseSignal: Noise — source deserialization/no-output error | e2e | 2 | 4.65%2 of 43 job runs affected | NoiseSignal: Noise — source deserialization/no-output error | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 3 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)/opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539718 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:41:58.528583" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:00.664
"ts"="2026-03-26 15:55:00.664343" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:04.464
"ts"="2026-03-26 15:55:04.464563" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:04.464576" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:04.464
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:56:17.712
"ts"="2026-03-26 15:56:17.712675" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-58q68d" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:15.153
"ts"="2026-03-26 16:07:15.153711" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:15.153724" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:16:41.008
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:17:18.424
STEP: capturing node release images before upgrade @ 03/26/26 16:17:18.558
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:17:18.709
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:18:14.022
STEP: verifying node pool GET still reflects the new version @ 03/26/26 16:46:24.898
"ts"="2026-03-26 16:46:25.999092" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:46:25.999204" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:26.755016" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"=""
"ts"="2026-03-26 16:46:28.395939" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396047" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396124" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:46:28.397295" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-124"
"ts"="2026-03-26 16:46:28.397316" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:28.601756" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:46:28.749306" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="msi-deployment-124"
"ts"="2026-03-26 16:46:28.873805" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="pooledNonMsiScopedAssignments-ad26qwpi75bc2"
"ts"="2026-03-26 16:46:29.618362" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx"
"ts"="2026-03-26 16:46:30.395419" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:46:30.395563" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:46:30.395689" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835068" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835095" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
Command Error: signal: killed
Deserializaion Error: no output from command
/opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539718 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:41:58.528583" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:00.664
"ts"="2026-03-26 15:55:00.664343" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:04.464
"ts"="2026-03-26 15:55:04.464563" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:04.464576" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:04.464
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:56:17.712
"ts"="2026-03-26 15:56:17.712675" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-58q68d" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:15.153
"ts"="2026-03-26 16:07:15.153711" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:15.153724" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:16:41.008
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:17:18.424
STEP: capturing node release images before upgrade @ 03/26/26 16:17:18.558
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:17:18.709
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:18:14.022
STEP: verifying node pool GET still reflects the new version @ 03/26/26 16:46:24.898
"ts"="2026-03-26 16:46:25.999092" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:46:25.999204" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:26.755016" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"=""
"ts"="2026-03-26 16:46:28.395939" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396047" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396124" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:46:28.397295" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-124"
"ts"="2026-03-26 16:46:28.397316" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:28.601756" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:46:28.749306" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="msi-deployment-124"
"ts"="2026-03-26 16:46:28.873805" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="pooledNonMsiScopedAssignments-ad26qwpi75bc2"
"ts"="2026-03-26 16:46:29.618362" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx"
"ts"="2026-03-26 16:46:30.395419" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:46:30.395563" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:46:30.395689" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835068" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835095" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
Command Error: signal: killed
Deserializaion Error: no output from commandRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539959 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 03/26/26 15:46:02.033
"ts"="2026-03-26 15:46:02.033400" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: creating cluster parameters @ 03/26/26 15:46:06.196
"ts"="2026-03-26 15:46:06.196672" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:46:06.196689" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:46:06.196
STEP: creating the HCP cluster @ 03/26/26 15:46:48.119
"ts"="2026-03-26 15:46:48.120027" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: getting credentials @ 03/26/26 15:58:00.443
STEP: creating the node pool with initial labels and taints @ 03/26/26 15:58:47.704
"ts"="2026-03-26 15:58:47.704646" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 15:58:47.704659" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: waiting for nodes to be ready @ 03/26/26 16:09:40.126
STEP: verifying initial labels are present on nodes @ 03/26/26 16:09:40.961
STEP: verifying initial taints are present on nodes @ 03/26/26 16:09:41.073
STEP: updating nodepool with new taints and scaling up @ 03/26/26 16:09:41.073
STEP: verifying nodes are scaled to the expected count @ 03/26/26 16:17:48.889
STEP: verifying new taint is present on newly created node @ 03/26/26 16:17:49.423
STEP: updating nodepool with a new label and scaling up @ 03/26/26 16:17:49.528
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:225 @ 03/26/26 17:02:49.529
"ts"="2026-03-26 17:02:49.529703" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 17:02:49.529828" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-dvwx88" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:50.522417" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-dvwx88" "resourceGroup"=""
"ts"="2026-03-26 17:02:52.051693" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051782" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051836" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 17:02:52.052935" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-25"
"ts"="2026-03-26 17:02:52.052955" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:52.249459" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 17:02:52.393486" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="msi-deployment-25"
"ts"="2026-03-26 17:02:52.538837" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
"ts"="2026-03-26 17:02:53.235631" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="customer-infra-cluster-np-labels-taints-dvwx88"
"ts"="2026-03-26 17:02:53.762868" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 17:02:53.762999" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 17:02:53.763130" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539959 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 03/26/26 15:46:02.033
"ts"="2026-03-26 15:46:02.033400" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: creating cluster parameters @ 03/26/26 15:46:06.196
"ts"="2026-03-26 15:46:06.196672" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:46:06.196689" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:46:06.196
STEP: creating the HCP cluster @ 03/26/26 15:46:48.119
"ts"="2026-03-26 15:46:48.120027" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: getting credentials @ 03/26/26 15:58:00.443
STEP: creating the node pool with initial labels and taints @ 03/26/26 15:58:47.704
"ts"="2026-03-26 15:58:47.704646" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 15:58:47.704659" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: waiting for nodes to be ready @ 03/26/26 16:09:40.126
STEP: verifying initial labels are present on nodes @ 03/26/26 16:09:40.961
STEP: verifying initial taints are present on nodes @ 03/26/26 16:09:41.073
STEP: updating nodepool with new taints and scaling up @ 03/26/26 16:09:41.073
STEP: verifying nodes are scaled to the expected count @ 03/26/26 16:17:48.889
STEP: verifying new taint is present on newly created node @ 03/26/26 16:17:49.423
STEP: updating nodepool with a new label and scaling up @ 03/26/26 16:17:49.528
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:225 @ 03/26/26 17:02:49.529
"ts"="2026-03-26 17:02:49.529703" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 17:02:49.529828" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-dvwx88" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:50.522417" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-dvwx88" "resourceGroup"=""
"ts"="2026-03-26 17:02:52.051693" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051782" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051836" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 17:02:52.052935" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-25"
"ts"="2026-03-26 17:02:52.052955" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:52.249459" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 17:02:52.393486" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="msi-deployment-25"
"ts"="2026-03-26 17:02:52.538837" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
"ts"="2026-03-26 17:02:53.235631" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="customer-infra-cluster-np-labels-taints-dvwx88"
"ts"="2026-03-26 17:02:53.762868" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 17:02:53.762999" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 17:02:53.763130" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
Command Error: signal: killed
Deserializaion Error: no output from commandRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539962 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:46:02.082189" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:02.492
"ts"="2026-03-26 15:55:02.492194" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:06.122
"ts"="2026-03-26 15:55:06.123025" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:06.123040" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:06.123
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:55:57.394
"ts"="2026-03-26 15:55:57.394964" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-25sqpq" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:52.506
"ts"="2026-03-26 16:07:52.506148" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:52.506162" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:19:27.978
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:20:38.475
STEP: capturing node release images before upgrade @ 03/26/26 16:20:38.598
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:20:38.808
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:21:44.788
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214 @ 03/26/26 16:51:44.79
"ts"="2026-03-26 16:51:44.790837" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:51:44.790942" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:45.965086" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"=""
"ts"="2026-03-26 16:51:48.588028" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588123" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588183" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:51:48.589294" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-49"
"ts"="2026-03-26 16:51:48.589305" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:48.745161" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:51:48.756385" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="pooledNonMsiScopedAssignments-7f6zxb5wg4rdw"
"ts"="2026-03-26 16:51:48.889592" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="msi-deployment-49"
"ts"="2026-03-26 16:51:49.447325" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph"
"ts"="2026-03-26 16:51:50.126755" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:51:50.126870" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:51:50.127003" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539962 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:46:02.082189" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:02.492
"ts"="2026-03-26 15:55:02.492194" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:06.122
"ts"="2026-03-26 15:55:06.123025" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:06.123040" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:06.123
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:55:57.394
"ts"="2026-03-26 15:55:57.394964" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-25sqpq" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:52.506
"ts"="2026-03-26 16:07:52.506148" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:52.506162" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:19:27.978
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:20:38.475
STEP: capturing node release images before upgrade @ 03/26/26 16:20:38.598
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:20:38.808
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:21:44.788
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214 @ 03/26/26 16:51:44.79
"ts"="2026-03-26 16:51:44.790837" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:51:44.790942" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:45.965086" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"=""
"ts"="2026-03-26 16:51:48.588028" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588123" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588183" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:51:48.589294" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-49"
"ts"="2026-03-26 16:51:48.589305" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:48.745161" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:51:48.756385" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="pooledNonMsiScopedAssignments-7f6zxb5wg4rdw"
"ts"="2026-03-26 16:51:48.889592" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="msi-deployment-49"
"ts"="2026-03-26 16:51:49.447325" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph"
"ts"="2026-03-26 16:51:50.126755" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:51:50.126870" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:51:50.127003" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
Command Error: signal: killed
Deserializaion Error: no output from commandContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscri...full failure pattern: ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 4.65%2 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 4 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc0014ce600>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-khck8x" in resourcegroup="customer-rg-2djx9g" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0014ce5e0>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001454ed0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc0014ce600>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-khck8x" in resourcegroup="customer-rg-2djx9g" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0014ce5e0>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001454ed0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0008cc0e0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-6jgfhl" in resourcegroup="rg-cluster-nsg-subnet-reuse-2l4lzn" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008cc0c0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e040f0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0008cc0e0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-6jgfhl" in resourcegroup="rg-cluster-nsg-subnet-reuse-2l4lzn" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008cc0c0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e040f0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc001568640>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-4jws2q" in resourcegroup="rg-cluster-nsg-subnet-reuse-czjmtq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001568620>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001484f00>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 2...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc001568640>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-4jws2q" in resourcegroup="rg-cluster-nsg-subnet-reuse-czjmtq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001568620>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001484f00>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 2...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 2h0m0s timeoutfull failure pattern: Process did not finish before 2h0m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 2 | 4.65%2 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-03-26T17:07:37Z"}Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
node pool upgrade verification failed: <node> (version 4full failure pattern: node pool upgrade verification failed: <node> (version 4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 4.65%2 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002fe2c0>:
node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-4hzfn","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-psqk4","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002fe2c0>:
node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-4hzfn","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-psqk4","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002ff7a0>:
node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdf391b97b311ecf8f0e5be97024014ad8b264d7785313081aa1290eaf0bc100","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fadd21095ba14bd0d033a74eacbd06f7e40a03b9acbaa846a85a5dc16d73ca00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aadc8e5d5141737eec06abcb21778ea74ceca2076ed31a2b1277f332d837d2f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9eb3bea4e623795326319ebb95624f9b2a0905aac10ced9e74260931aa33ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6afe7e59c25f2df8e44971d20bf90820ff5c66abac524e656b2f84121032126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-5c7rp","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-ws7nw","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002ff7a0>:
node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdf391b97b311ecf8f0e5be97024014ad8b264d7785313081aa1290eaf0bc100","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fadd21095ba14bd0d033a74eacbd06f7e40a03b9acbaa846a85a5dc16d73ca00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aadc8e5d5141737eec06abcb21778ea74ceca2076ed31a2b1277f332d837d2f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9eb3bea4e623795326319ebb95624f9b2a0905aac10ced9e74260931aa33ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6afe7e59c25f2df8e44971d20bf90820ff5c66abac524e656b2f84121032126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-5c7rp","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-ws7nw","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 4 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0015f6390>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-qh8cxr" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00096b6c0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b660>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b640>{
msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0015f6390>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-qh8cxr" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00096b6c0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b660>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b640>{
msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001447860>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-58tkbd" in resourcegroup="rg-np-version-upgrade-58tkbd-xfhc4s" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000943000>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942fc0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942f80>{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001447860>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-58tkbd" in resourcegroup="rg-np-version-upgrade-58tkbd-xfhc4s" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000943000>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942fc0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942f80>{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0012d3cf8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5dhhk8" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000d51ac0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51aa0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51a80>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0012d3cf8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5dhhk8" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000d51ac0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51aa0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51a80>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (4)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:234]: Unexpected error:
<*errors.joinError | 0xc001072750>:
invalid node pool version update: expected error but none occurred
updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot update resource while resource is \"accepted\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*errors.errorString | 0xc0009011e0>{
s: "invalid node pool version update: expected error but none occurred",
},
<*errors.errorString | 0xc00011c850>{
s: "updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:234]: Unexpected error:
<*errors.joinError | 0xc001072750>:
invalid node pool version update: expected error but none occurred
updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: Conflict
--------------------------------------------------------------------------------
{
"error": {
"code": "Conflict",
"message": "Cannot update resource while resource is \"accepted\"",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
}
}
--------------------------------------------------------------------------------
{
errs: [
<*errors.errorString | 0xc0009011e0>{
s: "invalid node pool version update: expected error but none occurred",
},
<*errors.errorString | 0xc00011c850>{
s: "updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n }\n}\n--------------------------------------------------------------------------------\n",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' is in state 'updating', can't update; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' is in state 'updating', can't update; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
<*exported.ResponseError | 0xc0008b8030>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Cluster '2p8ffo390i7ajslf1kb46n95odfmnnis' is in state 'updating', can't update"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["166"],
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["376"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6d5a9527-c9df-4f8e-a7d1-0f9ae7243296",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Wed, 25 Mar 2026 02:30:44 GMT",
],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"93c91233-2cc4-4bc0-bd21-094c583f7b38",
],
"X-Ms-Correlation-Request-Id": [
"9caa67fa-6588-455f-9379-cb02eb456b06",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T023044Z:9caa67fa-6588-455f-9379-cb02eb456b06",
],
"X-Msedge-Ref": [
"Ref A: 51C988CAFCBE4C59903E48B577EB0CF4 Ref B: CHI261140502040 Ref C: 2026-03-25T02:30:44Z",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000a0c600>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 112, 56, 102, 102, 111, 51, 57, 48, 105, 55, 97, 106, 115, 108, 102, 49, 107, 98, 52, 54, 110, 57, 53, 111, 100, 102, 109, 110, 110, 105, 115, 39, 32, 105, 115, 32, 105, 110, 32, 115, 116, 97, 116, 101, 32, 39, 117, 112, 100, 97, 116, 105, 110, 103, 39, 44, 32, 99, 97, 110, 39, 116, 32, 117, 112, 100, 97, 116, 101, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 166,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PATCH",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Accept": ["application/json"],
"Content-Length": ["124"],
"Content-Type": ["application/json"],
"User-Agent": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
<*exported.ResponseError | 0xc0008b8030>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Cluster '2p8ffo390i7ajslf1kb46n95odfmnnis' is in state 'updating', can't update"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["166"],
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["376"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6d5a9527-c9df-4f8e-a7d1-0f9ae7243296",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Wed, 25 Mar 2026 02:30:44 GMT",
],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"93c91233-2cc4-4bc0-bd21-094c583f7b38",
],
"X-Ms-Correlation-Request-Id": [
"9caa67fa-6588-455f-9379-cb02eb456b06",
],
"X-Ms-Routing-Request-Id": [
"UKSOUTH:20260325T023044Z:9caa67fa-6588-455f-9379-cb02eb456b06",
],
"X-Msedge-Ref": [
"Ref A: 51C988CAFCBE4C59903E48B577EB0CF4 Ref B: CHI261140502040 Ref C: 2026-03-25T02:30:44Z",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000a0c600>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 112, 56, 102, 102, 111, 51, 57, 48, 105, 55, 97, 106, 115, 108, 102, 49, 107, 98, 52, 54, 110, 57, 53, 111, 100, 102, 109, 110, 110, 105, 115, 39, 32, 105, 115, 32, 105, 110, 32, 115, 116, 97, 116, 101, 32, 39, 117, 112, 100, 97, 116, 105, 110, 103, 39, 44, 32, 99, 97, 110, 39, 116, 32, 117, 112, 100, 97, 116, 101, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 166,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PATCH",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "management.azure.com",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Accept": ["application/json"],
"Content-Length": ["124"],
"Content-Type": ["application/json"],
"User-Agent": [
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
found 1 managed resource groups left behind HCP clusters in cluster-listing-<id>full failure pattern: found 1 managed resource groups left behind HCP clusters in cluster-listing-<id> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc000b03530>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
{
errs: [
<*errors.errorString | 0xc0011cfd90>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc000b03530>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
{
errs: [
<*errors.errorString | 0xc0011cfd90>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqqfull failure pattern: found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc0019920d8>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq
{
errs: [
<*errors.errorString | 0xc0012612b0>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc0019920d8>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq
{
errs: [
<*errors.errorString | 0xc0012612b0>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
get logs -n openshift-ingress router-default-<id> (deployment router-default): Get "<url>full failure pattern: get logs -n openshift-ingress router-default-<id> (deployment router-default): Get "<url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 1 | STG | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:207]: Timed out after 600.004s.
router-default deployment logs should be fetchable
Expected success, but got an error:
<*fmt.wrapError | 0xc000a4c380>:
get logs -n openshift-ingress router-default-5b6969fd4-nk2ff (deployment router-default): Get "https://10.0.0.6:10250/containerLogs/openshift-ingress/router-default-5b6969fd4-nk2ff/router": EOF
{
msg: "get logs -n openshift-ingress router-default-5b6969fd4-nk2ff (deployment router-default): Get \"https://10.0.0.6:10250/containerLogs/openshift-ingress/router-default-5b6969fd4-nk2ff/router\": EOF",
err: <*errors.StatusError | 0xc000f437c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Get \"https://10.0.0.6:10250/containerLogs/openshift-ingress/router-default-5b6969fd4-nk2ff/router\": EOF",
Reason: "",
Details: nil,
Code: 500,
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:207]: Timed out after 600.004s.
router-default deployment logs should be fetchable
Expected success, but got an error:
<*fmt.wrapError | 0xc000a4c380>:
get logs -n openshift-ingress router-default-5b6969fd4-nk2ff (deployment router-default): Get "https://10.0.0.6:10250/containerLogs/openshift-ingress/router-default-5b6969fd4-nk2ff/router": EOF
{
msg: "get logs -n openshift-ingress router-default-5b6969fd4-nk2ff (deployment router-default): Get \"https://10.0.0.6:10250/containerLogs/openshift-ingress/router-default-5b6969fd4-nk2ff/router\": EOF",
err: <*errors.StatusError | 0xc000f437c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Get \"https://10.0.0.6:10250/containerLogs/openshift-ingress/router-default-5b6969fd4-nk2ff/router\": EOF",
Reason: "",
Details: nil,
Code: 500,
},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 1 · Mar 28: 0 · Mar 29: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
<*fmt.wrapErrors | 0xc000feecc0>:
failed waiting for hcpCluster="cluster-zstream-4-20-bfhfnd" in resourcegroup="rg-zstream-upgrade-4-20-d9cwmk" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-bfhfnd in resource group rg-zstream-upgrade-4-20-d9cwmk, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-bfhfnd\" in resourcegroup=\"rg-zstream-upgrade-4-20-d9cwmk\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-bfhfnd in resource group rg-zstream-upgrade-4-20-d9cwmk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001291a60>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-bfhfnd in resource group rg-zstream-upgrade-4-20-d9cwmk",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
<*fmt.wrapErrors | 0xc000feecc0>:
failed waiting for hcpCluster="cluster-zstream-4-20-bfhfnd" in resourcegroup="rg-zstream-upgrade-4-20-d9cwmk" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-bfhfnd in resource group rg-zstream-upgrade-4-20-d9cwmk, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-bfhfnd\" in resourcegroup=\"rg-zstream-upgrade-4-20-d9cwmk\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-bfhfnd in resource group rg-zstream-upgrade-4-20-d9cwmk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001291a60>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-bfhfnd in resource group rg-zstream-upgrade-4-20-d9cwmk",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>full failure pattern: tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:213]: Timed out after 120.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:201 with:
kubectl should execute successfully from authorized VM
Unexpected error:
<*errors.errorString | 0xc0009f2770>:
E0326 02:40:03.194264 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.207738 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.221081 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.504134 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.522643 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io
{
s: "E0326 02:40:03.194264 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.207738 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.221081 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.504134 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.522643 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nUnable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:213]: Timed out after 120.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:201 with:
kubectl should execute successfully from authorized VM
Unexpected error:
<*errors.errorString | 0xc0009f2770>:
E0326 02:40:03.194264 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.207738 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.221081 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.504134 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.522643 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io
{
s: "E0326 02:40:03.194264 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.207738 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.221081 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.504134 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.522643 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nUnable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io",
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
<*fmt.wrapError | 0xc000e5a200>:
route host was never found: Get "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc00111a8a0>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io",
Err: <*tls.CertificateVerificationError | 0xc00111a870>{
UnverifiedCertificates: [
{
Raw: [48, 130, 3, 251, 48, 130, 2, 227, 160, 3, 2, 1, 2, 2, 8, 92, 203, 189, 73, 220, 104, 43, 31, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 23, 13, 50, 55, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 249, 228, 29, 76, 100, 40, 200, 92, 162, 72, 142, 129, 115, 47, 12, 145, 183, 72, 240, 1, 5, 92, 53, 47, 31, 202, 10, 174, 62, 48, 18, 31, 123, 117, 140, 213, 164, 154, 238, 253, 232, 211, 79, 197, 28, 90, 200, 94, 224, 0, 70, 240, 119, 180, 27, 69, 87, 250, 57, 250, 182, 205, 211, 32, 26, 243, 182, 5, 144, 142, 58, 253, 89, 73, 124, 131, 53, 242, 32, 215, 113, 115, 161, 138, 70, 204, 131, 156, 64, 119, 133, 189, 247, 73, 99, 55, 100, 228, 240, 153, 178, 98, 32, 168, 45, 26, 44, 226, 43, 23, 138, 243, 25, 89, 222, 29, 108, 66, 218, 157, 9, 246, 107, 237, 142, 13, 120, 2, 51, 162, 36, 40, 113, 174, 6, 24, 142, 141, 102, 87, 120, 195, 146, 75, 183, 146, 189, 224, 113, 188, 226, 122, 101, 188, 79, 120, 110, 96, 64, 136, 125, 35, 111, 198, 186, 195, 167, 74, 193, 60, 173, 53, 229, 146, 21, 60, 205, 130, 8, 88, 103, 27, 128, 125, 81, 34, 171, 43, 156, 241, 130, 164, 77, 16, 73, 90, 75, 165, 203, 168, 125, 253, 236, 167, 195, 124, 242, 13, 239, 113, 109, 161, 225, 227, 92, 142, 4, 22, 72, 42, 37, 19, 217, 72, 151, 205, 233, 36, 126, 107, 7, 87, 232, 186, 158, 130, 205, 56, 29, 44, 197, 167, 114, 125, 3, 48, 242, 51, 4, 177, 248, 136, 183, 247, 237, 2, 3, 1, 0, 1, 163, 130, 1, 33, 48, 130, 1, 29, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 12, 97, 255, 11, 226, 234, 126, 212, 87, 16, 110, 229, 159, 103, 200, 181, 174, 217, 7, 108, 202, 95, 92, 201, 13, 142, 12, 237, 141, 170, 208, 199, 74, 93, 166, 35, 81, 212, 86, 158, 162, 22, 175, 168, 213, 148, 254, 131, 79, 212, 162, 214, 255, 45, 231, 63, 104, 146, 47, 51, 18, 187, 156, 37, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 156, 211, 165, 107, 137, 142, 107, 233, 193, 15, 4, 182, 14, 206, 111, 124, 197, 24, 131, 13, 156, 218, 0, 253, 11, 83, 182, 227, 29, 64, 114, 33, 130, 86, 230, 58, 170, 150, 224, 131, 197, 120, 88, 8, 23, 135, 212, 114, 126, 131, 34, 46, 40, 21, 179, 71, 152, 82, 176, 205, 106, 13, 229, 19, 48, 70, 6, 3, 85, 29, 17, 4, 63, 48, 61, 130, 59, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 104, 50, 112, 54, 114, 55, 107, 57, 101, 52, 106, 56, 97, 52, 112, 46, 114, 102, 56, 119, 46, 97, 117, 115, 116, 114, 97, 108, 105, 97, 101, 97, 115, 116, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 105, 111, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 111, 129, 198, 35, 161, 213, 26, 120, 1, 118, 171, 24, 218, 108, 27, 212, 193, 115, 224, 125, 114, 122, 57, 94, 2, 5, 206, 118, 98, 94, 138, 212, 7...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
<*fmt.wrapError | 0xc000e5a200>:
route host was never found: Get "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc00111a8a0>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io",
Err: <*tls.CertificateVerificationError | 0xc00111a870>{
UnverifiedCertificates: [
{
Raw: [48, 130, 3, 251, 48, 130, 2, 227, 160, 3, 2, 1, 2, 2, 8, 92, 203, 189, 73, 220, 104, 43, 31, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 23, 13, 50, 55, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 249, 228, 29, 76, 100, 40, 200, 92, 162, 72, 142, 129, 115, 47, 12, 145, 183, 72, 240, 1, 5, 92, 53, 47, 31, 202, 10, 174, 62, 48, 18, 31, 123, 117, 140, 213, 164, 154, 238, 253, 232, 211, 79, 197, 28, 90, 200, 94, 224, 0, 70, 240, 119, 180, 27, 69, 87, 250, 57, 250, 182, 205, 211, 32, 26, 243, 182, 5, 144, 142, 58, 253, 89, 73, 124, 131, 53, 242, 32, 215, 113, 115, 161, 138, 70, 204, 131, 156, 64, 119, 133, 189, 247, 73, 99, 55, 100, 228, 240, 153, 178, 98, 32, 168, 45, 26, 44, 226, 43, 23, 138, 243, 25, 89, 222, 29, 108, 66, 218, 157, 9, 246, 107, 237, 142, 13, 120, 2, 51, 162, 36, 40, 113, 174, 6, 24, 142, 141, 102, 87, 120, 195, 146, 75, 183, 146, 189, 224, 113, 188, 226, 122, 101, 188, 79, 120, 110, 96, 64, 136, 125, 35, 111, 198, 186, 195, 167, 74, 193, 60, 173, 53, 229, 146, 21, 60, 205, 130, 8, 88, 103, 27, 128, 125, 81, 34, 171, 43, 156, 241, 130, 164, 77, 16, 73, 90, 75, 165, 203, 168, 125, 253, 236, 167, 195, 124, 242, 13, 239, 113, 109, 161, 225, 227, 92, 142, 4, 22, 72, 42, 37, 19, 217, 72, 151, 205, 233, 36, 126, 107, 7, 87, 232, 186, 158, 130, 205, 56, 29, 44, 197, 167, 114, 125, 3, 48, 242, 51, 4, 177, 248, 136, 183, 247, 237, 2, 3, 1, 0, 1, 163, 130, 1, 33, 48, 130, 1, 29, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 12, 97, 255, 11, 226, 234, 126, 212, 87, 16, 110, 229, 159, 103, 200, 181, 174, 217, 7, 108, 202, 95, 92, 201, 13, 142, 12, 237, 141, 170, 208, 199, 74, 93, 166, 35, 81, 212, 86, 158, 162, 22, 175, 168, 213, 148, 254, 131, 79, 212, 162, 214, 255, 45, 231, 63, 104, 146, 47, 51, 18, 187, 156, 37, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 156, 211, 165, 107, 137, 142, 107, 233, 193, 15, 4, 182, 14, 206, 111, 124, 197, 24, 131, 13, 156, 218, 0, 253, 11, 83, 182, 227, 29, 64, 114, 33, 130, 86, 230, 58, 170, 150, 224, 131, 197, 120, 88, 8, 23, 135, 212, 114, 126, 131, 34, 46, 40, 21, 179, 71, 152, 82, 176, 205, 106, 13, 229, 19, 48, 70, 6, 3, 85, 29, 17, 4, 63, 48, 61, 130, 59, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 104, 50, 112, 54, 114, 55, 107, 57, 101, 52, 106, 56, 97, 52, 112, 46, 114, 102, 56, 119, 46, 97, 117, 115, 116, 114, 97, 108, 105, 97, 101, 97, 115, 116, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 105, 111, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 111, 129, 198, 35, 161, 213, 26, 120, 1, 118, 171, 24, 218, 108, 27, 212, 193, 115, 224, 125, 114, 122, 57, 94, 2, 5, 206, 118, 98, 94, 138, 212, 7...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||