CIHealth
<
Single Day: Mar 25▾
Last 7 Days Weekly: Mar 23 - Mar 29 Sprint: Mar 16 - Mar 29 Single Day: Mar 25
Custom time window
> Reset
View JSON API

Environment: PROD

Failure patterns: 4 · Failures matched: 5 · Total runs: 9 · Runs affected: 4

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e222.22%2 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsMar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 2 · Mar 25: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0004ec240>: 
    failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation "cluster-zstream-4-19-ttlq29" in resourcegroup="rg-zstream-upgrade-4-19-dmw26l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0004ec200>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00110c240>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Msedge-Ref": [
                            "Ref A: 2550DE417C1E449D8F0C03A8300EB8A9 Ref B: CHI261140502040 Ref C: 2026-03-25T08:38:47Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 08:38:48 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Content-Length": ["138"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/b10cf018-156c-4dbb-acc4-7a2bd061c929",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "1ca5abf8-7530-42ab-b8ed-908d89ca29c9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "WESTEUROPE:20260325T083849Z:bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["ap...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0004ec240>: 
    failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation "cluster-zstream-4-19-ttlq29" in resourcegroup="rg-zstream-upgrade-4-19-dmw26l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-ttlq29: failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0004ec200>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-ttlq29\" in resourcegroup=\"rg-zstream-upgrade-4-19-dmw26l\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dmw26l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ttlq29\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00110c240>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Msedge-Ref": [
                            "Ref A: 2550DE417C1E449D8F0C03A8300EB8A9 Ref B: CHI261140502040 Ref C: 2026-03-25T08:38:47Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 08:38:48 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Content-Length": ["138"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/b10cf018-156c-4dbb-acc4-7a2bd061c929",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "1ca5abf8-7530-42ab-b8ed-908d89ca29c9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "WESTEUROPE:20260325T083849Z:bd561fc2-0a61-425d-96b6-a7f36d7cc4ac",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["ap...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009b92a0>: 
    failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation "cluster-zstream-4-19-gs2t7t" in resourcegroup="rg-zstream-upgrade-4-19-gtmk6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009b9280>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ed84e0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "00a7a86a-0829-4052-a505-f4a3f7fbe820",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Envoy-Upstream-Service-Time": ["47"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["138"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260325T024457Z:205f05d7-f7ea-4800-9e3a-0982d1258690",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: B109C53224CE4681B497F6C095BB9092 Ref B: CHI261140510062 Ref C: 2026-03-25T02:44:56Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 02:44:56 GMT",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/b6b931c5-6267-442b-bbcc-c343b9c46261",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009b92a0>: 
    failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation "cluster-zstream-4-19-gs2t7t" in resourcegroup="rg-zstream-upgrade-4-19-gtmk6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-gs2t7t: failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009b9280>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-gs2t7t\" in resourcegroup=\"rg-zstream-upgrade-4-19-gtmk6w\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gtmk6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-gs2t7t\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ed84e0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "00a7a86a-0829-4052-a505-f4a3f7fbe820",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Envoy-Upstream-Service-Time": ["47"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["138"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260325T024457Z:205f05d7-f7ea-4800-9e3a-0982d1258690",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: B109C53224CE4681B497F6C095BB9092 Ref B: CHI261140510062 Ref C: 2026-03-25T02:44:56Z",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 02:44:56 GMT",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/b6b931c5-6267-442b-bbcc-c343b9c46261",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.192
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2036714173588574208
n/a2036624181424033792
ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:234]: Unexpected error:
    <*errors.joinError | 0xc001072750>: 
    invalid node pool version update: expected error but none occurred
    updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*errors.errorString | 0xc0009011e0>{
                s: "invalid node pool version update: expected error but none occurred",
            },
            <*errors.errorString | 0xc00011c850>{
                s: "updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:234]: Unexpected error:
    <*errors.joinError | 0xc001072750>: 
    invalid node pool version update: expected error but none occurred
    updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*errors.errorString | 0xc0009011e0>{
                s: "invalid node pool version update: expected error but none occurred",
            },
            <*errors.errorString | 0xc00011c850>{
                s: "updating immutable fields: expected error containing keywords 'forbidden field immutable', got: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-rq5kvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2036756543294148608
ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' is in state 'updating', can't update; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' is in state 'updating', can't update; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
    <*exported.ResponseError | 0xc0008b8030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Cluster '2p8ffo390i7ajslf1kb46n95odfmnnis' is in state 'updating', can't update"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["166"],
                "Expires": ["-1"],
                "X-Envoy-Upstream-Service-Time": ["376"],
                "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                "X-Cache": ["CONFIG_NOCACHE"],
                "Cache-Control": ["no-cache"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Operation-Identifier": [
                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6d5a9527-c9df-4f8e-a7d1-0f9ae7243296",
                ],
                "Strict-Transport-Security": [
                    "max-age=31536000; includeSubDomains",
                ],
                "X-Content-Type-Options": ["nosniff"],
                "Date": [
                    "Wed, 25 Mar 2026 02:30:44 GMT",
                ],
                "Pragma": ["no-cache"],
                "X-Ms-Request-Id": [
                    "93c91233-2cc4-4bc0-bd21-094c583f7b38",
                ],
                "X-Ms-Correlation-Request-Id": [
                    "9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Ms-Routing-Request-Id": [
                    "UKSOUTH:20260325T023044Z:9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Msedge-Ref": [
                    "Ref A: 51C988CAFCBE4C59903E48B577EB0CF4 Ref B: CHI261140502040 Ref C: 2026-03-25T02:30:44Z",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000a0c600>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 112, 56, 102, 102, 111, 51, 57, 48, 105, 55, 97, 106, 115, 108, 102, 49, 107, 98, 52, 54, 110, 57, 53, 111, 100, 102, 109, 110, 110, 105, 115, 39, 32, 105, 115, 32, 105, 110, 32, 115, 116, 97, 116, 101, 32, 39, 117, 112, 100, 97, 116, 105, 110, 103, 39, 44, 32, 99, 97, 110, 39, 116, 32, 117, 112, 100, 97, 116, 101, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 166,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "management.azure.com",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2025-12-23-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Accept": ["application/json"],
                    "Content-Length": ["124"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
            ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
    <*exported.ResponseError | 0xc0008b8030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Cluster '2p8ffo390i7ajslf1kb46n95odfmnnis' is in state 'updating', can't update"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["166"],
                "Expires": ["-1"],
                "X-Envoy-Upstream-Service-Time": ["376"],
                "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                "X-Cache": ["CONFIG_NOCACHE"],
                "Cache-Control": ["no-cache"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Operation-Identifier": [
                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6d5a9527-c9df-4f8e-a7d1-0f9ae7243296",
                ],
                "Strict-Transport-Security": [
                    "max-age=31536000; includeSubDomains",
                ],
                "X-Content-Type-Options": ["nosniff"],
                "Date": [
                    "Wed, 25 Mar 2026 02:30:44 GMT",
                ],
                "Pragma": ["no-cache"],
                "X-Ms-Request-Id": [
                    "93c91233-2cc4-4bc0-bd21-094c583f7b38",
                ],
                "X-Ms-Correlation-Request-Id": [
                    "9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Ms-Routing-Request-Id": [
                    "UKSOUTH:20260325T023044Z:9caa67fa-6588-455f-9379-cb02eb456b06",
                ],
                "X-Msedge-Ref": [
                    "Ref A: 51C988CAFCBE4C59903E48B577EB0CF4 Ref B: CHI261140502040 Ref C: 2026-03-25T02:30:44Z",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000a0c600>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 39, 50, 112, 56, 102, 102, 111, 51, 57, 48, 105, 55, 97, 106, 115, 108, 102, 49, 107, 98, 52, 54, 110, 57, 53, 111, 100, 102, 109, 110, 110, 105, 115, 39, 32, 105, 115, 32, 105, 110, 32, 115, 116, 97, 116, 101, 32, 39, 117, 112, 100, 97, 116, 105, 110, 103, 39, 44, 32, 99, 97, 110, 39, 116, 32, 117, 112, 100, 97, 116, 101, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 166,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "management.azure.com",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vndvq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2025-12-23-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Accept": ["application/json"],
                    "Content-Length": ["124"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
            ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2036624181424033792
found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
full failure pattern:
found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 19: 0 · Mar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc000b03530>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
    {
        errs: [
            <*errors.errorString | 0xc0011cfd90>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc000b03530>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8
    {
        errs: [
            <*errors.errorString | 0xc0011cfd90>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-dzjxg8",
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to list HCP clusters without node pools at the resource group level1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2036778322305748992