CIHealth
<
Single Day: Mar 24▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:47:25 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009009c0>: 
    failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation "cluster-zstream-4-19-q94v6k" in resourcegroup="rg-zstream-upgrade-4-19-9pkbw7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000900980>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012389c0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "f19d9e44-6f27-4377-a922-81ce37e678a2",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260325T001907Z:f19d9e44-6f27-4377-a922-81ce37e678a2",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Length": ["138"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/58824f7e-b158-49e9-8f26-b1f9ce14eace",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 00:19:06 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "7087b8c2-6c1c-4159-98ea-9c1c04639dc3",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 86F1E95FE862488BABE9E655ECC14C19 Ref B: CHI261140501060 Ref C: 2026-03-25T00:19:06Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Type"...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009009c0>: 
    failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation "cluster-zstream-4-19-q94v6k" in resourcegroup="rg-zstream-upgrade-4-19-9pkbw7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-q94v6k: failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000900980>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-q94v6k\" in resourcegroup=\"rg-zstream-upgrade-4-19-9pkbw7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9pkbw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-q94v6k\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012389c0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "f19d9e44-6f27-4377-a922-81ce37e678a2",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260325T001907Z:f19d9e44-6f27-4377-a922-81ce37e678a2",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Length": ["138"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/58824f7e-b158-49e9-8f26-b1f9ce14eace",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 00:19:06 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "7087b8c2-6c1c-4159-98ea-9c1c04639dc3",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 86F1E95FE862488BABE9E655ECC14C19 Ref B: CHI261140501060 Ref C: 2026-03-25T00:19:06Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Type"...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
22:59:53 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000e72b00>: 
    failed to create HCP cluster cluster-zstream-4-19-2n9mwt: failed starting cluster creation "cluster-zstream-4-19-2n9mwt" in resourcegroup="rg-zstream-upgrade-4-19-mbbz5p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-mbbz5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-2n9mwt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-2n9mwt: failed starting cluster creation \"cluster-zstream-4-19-2n9mwt\" in resourcegroup=\"rg-zstream-upgrade-4-19-mbbz5p\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-mbbz5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-2n9mwt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e72ae0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-2n9mwt\" in resourcegroup=\"rg-zstream-upgrade-4-19-mbbz5p\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-mbbz5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-2n9mwt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010a9800>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["53"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "98b23134-d33f-4813-9b8a-67822cc108fb",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "SWITZERLANDNORTH:20260324T230853Z:98b23134-d33f-4813-9b8a-67822cc108fb",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Tue, 24 Mar 2026 23:08:52 GMT",
                        ],
                        "Content-Length": ["138"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "6d01c108-abb5-42cc-959d-f73f412ce62c",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: C4C85871B5364069B3167339E6E0127B Ref B: CH1AA2020617021 Ref C: 2026-03-24T23:08:52Z",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/0d980817-2a03-49a2-b04b-0accb82b0a1e",
                        ],
                        "Strict-Transport-Security": [...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000e72b00>: 
    failed to create HCP cluster cluster-zstream-4-19-2n9mwt: failed starting cluster creation "cluster-zstream-4-19-2n9mwt" in resourcegroup="rg-zstream-upgrade-4-19-mbbz5p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-mbbz5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-2n9mwt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-2n9mwt: failed starting cluster creation \"cluster-zstream-4-19-2n9mwt\" in resourcegroup=\"rg-zstream-upgrade-4-19-mbbz5p\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-mbbz5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-2n9mwt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e72ae0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-2n9mwt\" in resourcegroup=\"rg-zstream-upgrade-4-19-mbbz5p\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-mbbz5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-2n9mwt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010a9800>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["53"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "98b23134-d33f-4813-9b8a-67822cc108fb",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "SWITZERLANDNORTH:20260324T230853Z:98b23134-d33f-4813-9b8a-67822cc108fb",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Tue, 24 Mar 2026 23:08:52 GMT",
                        ],
                        "Content-Length": ["138"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "6d01c108-abb5-42cc-959d-f73f412ce62c",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: C4C85871B5364069B3167339E6E0127B Ref B: CH1AA2020617021 Ref C: 2026-03-24T23:08:52Z",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/0d980817-2a03-49a2-b04b-0accb82b0a1e",
                        ],
                        "Strict-Transport-Security": [...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a