CIHealth
<
Single Day: Mar 25▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
12:12:46 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at the resource group level · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc000b03530>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
    {
        errs: [
            <*errors.errorString | 0xc0011cfd90>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc000b03530>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
    {
        errs: [
            <*errors.errorString | 0xc0011cfd90>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8",
            },
        ],
    }
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift
10:46:15 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:234]: Unexpected error:
    <*errors.joinError | 0xc001072750>: 
    invalid node pool version update: expected error but none occurred
    updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*errors.errorString | 0xc0009011e0>{
                s: "invalid node pool version update: expected error but none occurred",
            },
            <*errors.errorString | 0xc00011c850>{
                s: "updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:234]: Unexpected error:
    <*errors.joinError | 0xc001072750>: 
    invalid node pool version update: expected error but none occurred
    updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*errors.errorString | 0xc0009011e0>{
                s: "invalid node pool version update: expected error but none occurred",
            },
            <*errors.errorString | 0xc00011c850>{
                s: "updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        ],
    }
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
07:57:53 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0004ec240>: 
    failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation "cluster-zstream-4-19-ttlq29" in resourcegroup="rg-zstream-upgrade-4-19-dmw26l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0004ec200>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00110c240>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Msedge-Ref": [
                            "Ref A: 2550DE417C1E449D8F0C03A8300EB8A9 Ref B: CHI261140502040 Ref C: 2026-03-25T08:38:47Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 08:38:48 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Content-Length": ["138"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/b10cf018-156c-4dbb-acc4-7a2bd061c929",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "1ca5abf8-7530-42ab-b8ed-908d89ca29c9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "WESTEUROPE:20260325T083849Z:bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["ap...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0004ec240>: 
    failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation "cluster-zstream-4-19-ttlq29" in resourcegroup="rg-zstream-upgrade-4-19-dmw26l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0004ec200>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00110c240>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Msedge-Ref": [
                            "Ref A: 2550DE417C1E449D8F0C03A8300EB8A9 Ref B: CHI261140502040 Ref C: 2026-03-25T08:38:47Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 08:38:48 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Content-Length": ["138"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/b10cf018-156c-4dbb-acc4-7a2bd061c929",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "1ca5abf8-7530-42ab-b8ed-908d89ca29c9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "WESTEUROPE:20260325T083849Z:bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["ap...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
02:00:18 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009b92a0>: 
    failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation "cluster-zstream-4-19-gs2t7t" in resourcegroup="rg-zstream-upgrade-4-19-gtmk6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009b9280>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ed84e0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "00a7a86a-0829-4052-a505-f4a3f7fbe820",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Envoy-Upstream-Service-Time": ["47"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["138"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260325T024457Z:205f05d7-f7ea-4800-9e3a-0982d1258690",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: B109C53224CE4681B497F6C095BB9092 Ref B: CHI261140510062 Ref C: 2026-03-25T02:44:56Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 02:44:56 GMT",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/b6b931c5-6267-442b-bbcc-c343b9c46261",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009b92a0>: 
    failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation "cluster-zstream-4-19-gs2t7t" in resourcegroup="rg-zstream-upgrade-4-19-gtmk6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009b9280>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ed84e0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "00a7a86a-0829-4052-a505-f4a3f7fbe820",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Envoy-Upstream-Service-Time": ["47"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["138"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260325T024457Z:205f05d7-f7ea-4800-9e3a-0982d1258690",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: B109C53224CE4681B497F6C095BB9092 Ref B: CHI261140510062 Ref C: 2026-03-25T02:44:56Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 02:44:56 GMT",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/b6b931c5-6267-442b-bbcc-c343b9c46261",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' is in state 'updating', can't update; provider Microsoft.RedHatOpenShift
02:00:18 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
    <*exported.ResponseError | 0xc0008b8030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Cluster '2p8ffo390i7ajslf1kb46n95odfmnnis' is in state 'updating', can't update"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["166"],
                "Expires": ["-1"],
                "X-Envoy-Upstream-Service-Time": ["376"],
                "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                "X-Cache": ["CONFIG_NOCACHE"],
                "Cache-Control": ["no-cache"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Operation-Identifier": [
                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6d5a9527-c9df-4f8e-a7d1-0f9ae7243296",
                ],
                "Strict-Transport-Security": [
                    "max-age=31536000; includeSubDomains",
                ],
                "X-Content-Type-Options": ["nosniff"],
                "Date": [
                    "Wed, 25 Mar 2026 02:30:44 GMT",
                ],
                "Pragma": ["no-cache"],
                "X-Ms-Request-Id": [
                    "93c91233-2cc4-4bc0-bd21-094c583f7b38",
                ],
                "X-Ms-Correlation-Request-Id": [
                    "9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Ms-Routing-Request-Id": [
                    "UKSOUTH:20260325T023044Z:9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Msedge-Ref": [
                    "Ref A: 51C988CAFCBE4C59903E48B577EB0CF4 Ref B: CHI261140502040 Ref C: 2026-03-25T02:30:44Z",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000a0c600>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 112, 56, 102, 102, 111, 51, 57, 48, 105, 55, 97, 106, 115, 108, 102, 49, 107, 98, 52, 54, 110, 57, 53, 111, 100, 102, 109, 110, 110, 105, 115, 39, 32, 105, 115, 32, 105, 110, 32, 115, 116, 97, 116, 101, 32, 39, 117, 112, 100, 97, 116, 105, 110, 103, 39, 44, 32, 99, 97, 110, 39, 116, 32, 117, 112, 100, 97, 116, 101, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 166,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "management.azure.com",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2025-12-23-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Accept": ["application/json"],
                    "Content-Length": ["124"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
            ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
    <*exported.ResponseError | 0xc0008b8030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Cluster '2p8ffo390i7ajslf1kb46n95odfmnnis' is in state 'updating', can't update"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["166"],
                "Expires": ["-1"],
                "X-Envoy-Upstream-Service-Time": ["376"],
                "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                "X-Cache": ["CONFIG_NOCACHE"],
                "Cache-Control": ["no-cache"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Operation-Identifier": [
                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6d5a9527-c9df-4f8e-a7d1-0f9ae7243296",
                ],
                "Strict-Transport-Security": [
                    "max-age=31536000; includeSubDomains",
                ],
                "X-Content-Type-Options": ["nosniff"],
                "Date": [
                    "Wed, 25 Mar 2026 02:30:44 GMT",
                ],
                "Pragma": ["no-cache"],
                "X-Ms-Request-Id": [
                    "93c91233-2cc4-4bc0-bd21-094c583f7b38",
                ],
                "X-Ms-Correlation-Request-Id": [
                    "9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Ms-Routing-Request-Id": [
                    "UKSOUTH:20260325T023044Z:9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Msedge-Ref": [
                    "Ref A: 51C988CAFCBE4C59903E48B577EB0CF4 Ref B: CHI261140502040 Ref C: 2026-03-25T02:30:44Z",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000a0c600>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 112, 56, 102, 102, 111, 51, 57, 48, 105, 55, 97, 106, 115, 108, 102, 49, 107, 98, 52, 54, 110, 57, 53, 111, 100, 102, 109, 110, 110, 105, 115, 39, 32, 105, 115, 32, 105, 110, 32, 115, 116, 97, 116, 101, 32, 39, 117, 112, 100, 97, 116, 105, 110, 103, 39, 44, 32, 99, 97, 110, 39, 116, 32, 117, 112, 100, 97, 116, 101, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 166,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "management.azure.com",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2025-12-23-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Accept": ["application/json"],
                    "Content-Length": ["124"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
            ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred