<
Single Day: Mar 24▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0ea6d9b · merge 0ea6d9b
n/a e2e failed ★#4614 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:49:32 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0006966a0>: 
    failed to create HCP cluster cluster-zstream-4-19-rj49ln: failed starting cluster creation "cluster-zstream-4-19-rj49ln" in resourcegroup="rg-zstream-upgrade-4-19-pc2gx9": PUT https://rp.j4029440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-pc2gx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-rj49ln
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-rj49ln: failed starting cluster creation \"cluster-zstream-4-19-rj49ln\" in resourcegroup=\"rg-zstream-upgrade-4-19-pc2gx9\": PUT https://rp.j4029440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-pc2gx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-rj49ln\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000696680>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-rj49ln\" in resourcegroup=\"rg-zstream-upgrade-4-19-pc2gx9\": PUT https://rp.j4029440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-pc2gx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-rj49ln\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00063aab0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "38dba7f1-fd4e-420c-a529-1198305aac8e",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 00:50:44 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["45"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0006965a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0006966a0>: 
    failed to create HCP cluster cluster-zstream-4-19-rj49ln: failed starting cluster creation "cluster-zstream-4-19-rj49ln" in resourcegroup="rg-zstream-upgrade-4-19-pc2gx9": PUT https://rp.j4029440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-pc2gx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-rj49ln
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-rj49ln: failed starting cluster creation \"cluster-zstream-4-19-rj49ln\" in resourcegroup=\"rg-zstream-upgrade-4-19-pc2gx9\": PUT https://rp.j4029440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-pc2gx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-rj49ln\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000696680>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-rj49ln\" in resourcegroup=\"rg-zstream-upgrade-4-19-pc2gx9\": PUT https://rp.j4029440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-pc2gx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-rj49ln\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00063aab0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "38dba7f1-fd4e-420c-a529-1198305aac8e",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 00:50:44 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["45"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0006965a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fa604f2
n/a e2e failed ★#4603 (open) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:24:40 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000c02240>: 
    failed to create HCP cluster cluster-zstream-4-19-267c5s: failed starting cluster creation "cluster-zstream-4-19-267c5s" in resourcegroup="rg-zstream-upgrade-4-19-qmdpmw": PUT https://rp.j5240832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-qmdpmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-267c5s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-267c5s: failed starting cluster creation \"cluster-zstream-4-19-267c5s\" in resourcegroup=\"rg-zstream-upgrade-4-19-qmdpmw\": PUT https://rp.j5240832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-qmdpmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-267c5s\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c02220>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-267c5s\" in resourcegroup=\"rg-zstream-upgrade-4-19-qmdpmw\": PUT https://rp.j5240832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-qmdpmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-267c5s\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010762a0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["46"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "5f4aaf79-a99d-4a76-90aa-286a3071d601",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 00:29:59 GMT",
                        ],
                        "Content-Length": ["138"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000c02160>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000c02240>: 
    failed to create HCP cluster cluster-zstream-4-19-267c5s: failed starting cluster creation "cluster-zstream-4-19-267c5s" in resourcegroup="rg-zstream-upgrade-4-19-qmdpmw": PUT https://rp.j5240832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-qmdpmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-267c5s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-267c5s: failed starting cluster creation \"cluster-zstream-4-19-267c5s\" in resourcegroup=\"rg-zstream-upgrade-4-19-qmdpmw\": PUT https://rp.j5240832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-qmdpmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-267c5s\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c02220>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-267c5s\" in resourcegroup=\"rg-zstream-upgrade-4-19-qmdpmw\": PUT https://rp.j5240832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-qmdpmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-267c5s\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010762a0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["46"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "5f4aaf79-a99d-4a76-90aa-286a3071d601",
                        ],
                        "Date": [
                            "Wed, 25 Mar 2026 00:29:59 GMT",
                        ],
                        "Content-Length": ["138"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000c02160>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 87a94b6 · merge e4ce686
n/a e2e failed ★#4539 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:00:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc00093d4a0>: 
    failed to create HCP cluster cluster-zstream-4-19-zbgbs7: failed starting cluster creation "cluster-zstream-4-19-zbgbs7" in resourcegroup="rg-zstream-upgrade-4-19-jfx54m": PUT https://rp.j7554816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jfx54m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zbgbs7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-zbgbs7: failed starting cluster creation \"cluster-zstream-4-19-zbgbs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-jfx54m\": PUT https://rp.j7554816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jfx54m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zbgbs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00093d460>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-zbgbs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-jfx54m\": PUT https://rp.j7554816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jfx54m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zbgbs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00145aae0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "376dae8c-016c-44fa-8968-1e73e43961ce",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:49:01 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["88"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc00093d380>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc00093d4a0>: 
    failed to create HCP cluster cluster-zstream-4-19-zbgbs7: failed starting cluster creation "cluster-zstream-4-19-zbgbs7" in resourcegroup="rg-zstream-upgrade-4-19-jfx54m": PUT https://rp.j7554816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jfx54m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zbgbs7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-zbgbs7: failed starting cluster creation \"cluster-zstream-4-19-zbgbs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-jfx54m\": PUT https://rp.j7554816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jfx54m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zbgbs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00093d460>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-zbgbs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-jfx54m\": PUT https://rp.j7554816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jfx54m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zbgbs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00145aae0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "376dae8c-016c-44fa-8968-1e73e43961ce",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:49:01 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["88"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc00093d380>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 8df604b · merge 0ea6d9b
n/a provision failed ⚠#4614 (merged) 1
provision (1)
ERROR CODE: DeploymentNotFound
23:00:02 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/auditLogs/output · suite step graph
Show raw failure
time=2026-03-24T23:35:04.558Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=auditLogs step=output description="Step output\n  Kind: ARM\n  Template: templates/output-audit-logs-eventhub.bicep\n  Parameters: configurations/output-audit-logs-eventhub.tmpl.bicepparam"
time=2026-03-24T23:35:04.967Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=auditLogs step=output
time=2026-03-24T23:35:05.542Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=auditLogs step=output deployment=1468310f1bc0468f9ed7b0073890fee99bf8c2fff75858f2cf229991d137607d
time=2026-03-24T23:35:05.624Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=auditLogs step=output err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/westus3-shared-resources/providers/Microsoft.Resources/deployments/1468310f1bc0468f9ed7b0073890fee99bf8c2fff75858f2cf229991d137607d/operationStatuses/08584272115801546536\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment '1468310f1bc0468f9ed7b0073890fee99bf8c2fff75858f2cf229991d137607d' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head afaa40f · merge 582667c
n/a e2e failed ★#4588 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
22:30:09 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0008c5780>: 
    failed to create HCP cluster cluster-zstream-4-19-246j5l: failed starting cluster creation "cluster-zstream-4-19-246j5l" in resourcegroup="rg-zstream-upgrade-4-19-dk77vq": PUT https://rp.j4090240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dk77vq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-246j5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-246j5l: failed starting cluster creation \"cluster-zstream-4-19-246j5l\" in resourcegroup=\"rg-zstream-upgrade-4-19-dk77vq\": PUT https://rp.j4090240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dk77vq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-246j5l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008c5760>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-246j5l\" in resourcegroup=\"rg-zstream-upgrade-4-19-dk77vq\": PUT https://rp.j4090240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dk77vq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-246j5l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00131cbd0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "7b77ae63-ce18-45be-b106-3c6176e301eb",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:38:23 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["50"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0008c5660>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0008c5780>: 
    failed to create HCP cluster cluster-zstream-4-19-246j5l: failed starting cluster creation "cluster-zstream-4-19-246j5l" in resourcegroup="rg-zstream-upgrade-4-19-dk77vq": PUT https://rp.j4090240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dk77vq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-246j5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-246j5l: failed starting cluster creation \"cluster-zstream-4-19-246j5l\" in resourcegroup=\"rg-zstream-upgrade-4-19-dk77vq\": PUT https://rp.j4090240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dk77vq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-246j5l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008c5760>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-246j5l\" in resourcegroup=\"rg-zstream-upgrade-4-19-dk77vq\": PUT https://rp.j4090240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-dk77vq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-246j5l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00131cbd0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "7b77ae63-ce18-45be-b106-3c6176e301eb",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:38:23 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["50"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0008c5660>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 8914600
n/a provision failed #4612 (closed) 1
provision (1)
make[2]: *** [Makefile:32: deploy] Error 1
22:29:10 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph
Show raw failure
time=2026-03-24T23:09:24.452Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n  Kind: Shell\n  Command: make deploy\n"
time=2026-03-24T23:19:44.737Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json | kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=bba9ecaf-0a5e-47af-8d23-c33286ffeb31 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=f3698ec3-5fdb-4fee-8a48-711baac420de && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3852928.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3852928.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3852928 -n arohcp-prow-dbcs-j3852928 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3852928/providers/Microsoft.Network/dnszones/j3852928.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3852928-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3852928-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n  -f helm-charts/cluster-service/${OVERRIDES} \\\n  --set serviceAccountName=clusters-service \\\n  --set environment=arohcpprow \\\n  --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n  --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n  --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n  --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n  --set tlsCertificatesEnabled=true \\\n  --set denyAssignments=disabled \\\n  --set tenantId=${TENANT_ID} \\\n  --set region=westus3 \\\n  --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n  --set replicas=3 \\\n  --set imageRegistry=arohcpsvcdev.azurecr.io \\\n  --set imageRepository=app-sre/aro-hcp-clusters-service \\\n  --set imageDigest=sha256:5f079a0240687e6d2214d892a61019db3a3bbb4e3afb97bcf7731633af4ce7d8 \\\n  --set global.imageDigest=sha256:5f079a0240687e6d2214d892a61019db3a3bbb4e3afb97bcf7731633af4ce7d8 \\\n  --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n  --set batchProcessesDryRun=true \\\n  --set batchProcesses=\"\" \\\n  --set fpaCertName=firstPartyCert2 \\\n  --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n  --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n  --set shard.consumerName=\"hcp-underlay-j3852928-mgmt-1\" \\\n  --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n  --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n  --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n  --set shard.cxSecretsKeyVaultMiClientId=\"9a47fa51-2376-4a6c-b822-670de4dd981b\" \\\n  --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n  --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n  --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3852928-mgmt-1'))\") \\\n  --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3852928-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3852928-mgmt-1 \\\n  --set databaseHost=${DB_HOST} \\\n  --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n  --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n  --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n  --set azureArmHelperIdentityCertName=armHelperCert2 \\\n  --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n  --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n  --set azureCloudEnvironmentName=AzurePublicCloud \\\n  --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n  --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n  --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n  --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n  --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n  --set tracing.address=http://ingest.observability:4318 \\\n  --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n  --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n  --set provisionShardClusterLimit=100 \\\n  --set ocpVersions.defaultVersion.version=4.20.8 \\\n  --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n  --set ocpVersions.channelGroups.stable.maxVersion= \\\n  --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n  --set ocpVersions.channelGroups.candidate.maxVersion= \\\n  --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n  --set ocpVersions.channelGroups.nightly.maxVersion= \\\n  --set azureOperatorsMI.roleSetName=dev \\\n  --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: context deadline exceeded\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9e233cc
n/a e2e failed ★#4556 (open) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
22:27:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000493660>: 
    failed to create HCP cluster cluster-zstream-4-19-nzpzxv: failed starting cluster creation "cluster-zstream-4-19-nzpzxv" in resourcegroup="rg-zstream-upgrade-4-19-z99gwj": PUT https://rp.j6091520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-z99gwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nzpzxv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-nzpzxv: failed starting cluster creation \"cluster-zstream-4-19-nzpzxv\" in resourcegroup=\"rg-zstream-upgrade-4-19-z99gwj\": PUT https://rp.j6091520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-z99gwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nzpzxv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000493640>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-nzpzxv\" in resourcegroup=\"rg-zstream-upgrade-4-19-z99gwj\": PUT https://rp.j6091520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-z99gwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nzpzxv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00140a570>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "7955bfc9-e142-4be0-830d-7b198283f84d",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:32:17 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["40"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000493460>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000493660>: 
    failed to create HCP cluster cluster-zstream-4-19-nzpzxv: failed starting cluster creation "cluster-zstream-4-19-nzpzxv" in resourcegroup="rg-zstream-upgrade-4-19-z99gwj": PUT https://rp.j6091520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-z99gwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nzpzxv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-nzpzxv: failed starting cluster creation \"cluster-zstream-4-19-nzpzxv\" in resourcegroup=\"rg-zstream-upgrade-4-19-z99gwj\": PUT https://rp.j6091520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-z99gwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nzpzxv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000493640>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-nzpzxv\" in resourcegroup=\"rg-zstream-upgrade-4-19-z99gwj\": PUT https://rp.j6091520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-z99gwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-nzpzxv\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00140a570>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "7955bfc9-e142-4be0-830d-7b198283f84d",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:32:17 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["40"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000493460>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 70aec38 · merge 445d308
n/a e2e failed ★#3973 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
22:16:03 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000e145a0>: 
    failed to create HCP cluster cluster-zstream-4-19-dclbd6: failed starting cluster creation "cluster-zstream-4-19-dclbd6" in resourcegroup="rg-zstream-upgrade-4-19-jcrbds": PUT https://rp.j9519488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jcrbds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-dclbd6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-dclbd6: failed starting cluster creation \"cluster-zstream-4-19-dclbd6\" in resourcegroup=\"rg-zstream-upgrade-4-19-jcrbds\": PUT https://rp.j9519488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jcrbds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-dclbd6\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e14580>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-dclbd6\" in resourcegroup=\"rg-zstream-upgrade-4-19-jcrbds\": PUT https://rp.j9519488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jcrbds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-dclbd6\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012a0c30>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "c75630de-133b-4d88-b74b-a3488c538c90",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:21:11 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["79"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000e144c0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000e145a0>: 
    failed to create HCP cluster cluster-zstream-4-19-dclbd6: failed starting cluster creation "cluster-zstream-4-19-dclbd6" in resourcegroup="rg-zstream-upgrade-4-19-jcrbds": PUT https://rp.j9519488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jcrbds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-dclbd6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-dclbd6: failed starting cluster creation \"cluster-zstream-4-19-dclbd6\" in resourcegroup=\"rg-zstream-upgrade-4-19-jcrbds\": PUT https://rp.j9519488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jcrbds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-dclbd6\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e14580>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-dclbd6\" in resourcegroup=\"rg-zstream-upgrade-4-19-jcrbds\": PUT https://rp.j9519488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-jcrbds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-dclbd6\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012a0c30>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "c75630de-133b-4d88-b74b-a3488c538c90",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 23:21:11 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["79"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000e144c0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 364fb5c · merge 364fb5c
n/a e2e failed ★#4541 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
21:33:07 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000a1bbe0>: 
    failed to create HCP cluster cluster-zstream-4-19-pghf64: failed starting cluster creation "cluster-zstream-4-19-pghf64" in resourcegroup="rg-zstream-upgrade-4-19-cdxzwd": PUT https://rp.j1299712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cdxzwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pghf64
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-pghf64: failed starting cluster creation \"cluster-zstream-4-19-pghf64\" in resourcegroup=\"rg-zstream-upgrade-4-19-cdxzwd\": PUT https://rp.j1299712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cdxzwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pghf64\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a1bbc0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-pghf64\" in resourcegroup=\"rg-zstream-upgrade-4-19-cdxzwd\": PUT https://rp.j1299712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cdxzwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pghf64\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00139dbf0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "c431ec20-b434-44c3-9361-0ee984e17c83",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:43:35 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["45"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a1bac0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000a1bbe0>: 
    failed to create HCP cluster cluster-zstream-4-19-pghf64: failed starting cluster creation "cluster-zstream-4-19-pghf64" in resourcegroup="rg-zstream-upgrade-4-19-cdxzwd": PUT https://rp.j1299712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cdxzwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pghf64
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-pghf64: failed starting cluster creation \"cluster-zstream-4-19-pghf64\" in resourcegroup=\"rg-zstream-upgrade-4-19-cdxzwd\": PUT https://rp.j1299712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cdxzwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pghf64\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a1bbc0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-pghf64\" in resourcegroup=\"rg-zstream-upgrade-4-19-cdxzwd\": PUT https://rp.j1299712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cdxzwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-pghf64\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00139dbf0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "c431ec20-b434-44c3-9361-0ee984e17c83",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:43:35 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["45"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a1bac0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fd99db3
n/a e2e failed ★#4515 (closed) 2
e2e (2)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
21:24:10 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000c8e9c0>: 
    failed to create HCP cluster cluster-zstream-4-19-fksgqz: failed starting cluster creation "cluster-zstream-4-19-fksgqz" in resourcegroup="rg-zstream-upgrade-4-19-gxztpf": PUT https://rp.j1733376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gxztpf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-fksgqz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-fksgqz: failed starting cluster creation \"cluster-zstream-4-19-fksgqz\" in resourcegroup=\"rg-zstream-upgrade-4-19-gxztpf\": PUT https://rp.j1733376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gxztpf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-fksgqz\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c8e9a0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-fksgqz\" in resourcegroup=\"rg-zstream-upgrade-4-19-gxztpf\": PUT https://rp.j1733376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gxztpf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-fksgqz\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001274660>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "18a1547c-f43f-4c50-a88c-24b66dbe1e3a",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:36:41 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["1046"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000c8e8a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000c8e9c0>: 
    failed to create HCP cluster cluster-zstream-4-19-fksgqz: failed starting cluster creation "cluster-zstream-4-19-fksgqz" in resourcegroup="rg-zstream-upgrade-4-19-gxztpf": PUT https://rp.j1733376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gxztpf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-fksgqz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-fksgqz: failed starting cluster creation \"cluster-zstream-4-19-fksgqz\" in resourcegroup=\"rg-zstream-upgrade-4-19-gxztpf\": PUT https://rp.j1733376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gxztpf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-fksgqz\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c8e9a0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-fksgqz\" in resourcegroup=\"rg-zstream-upgrade-4-19-gxztpf\": PUT https://rp.j1733376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-gxztpf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-fksgqz\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001274660>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "18a1547c-f43f-4c50-a88c-24b66dbe1e3a",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:36:41 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["1046"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000c8e8a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Customer should be able to use workload identity via the cluster OIDC issuer URL: Timed out after <duration>s.
21:24:10 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:276]: Timed out after 300.000s.
Expected
    <v1.PodPhase>: Running
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:276]: Timed out after 300.000s.
Expected
    <v1.PodPhase>: Running
to equal
    <v1.PodPhase>: Succeeded
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9baa329 · merge 9baa329
n/a e2e failed ★#4461 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
21:23:14 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0007e2680>: 
    failed to create HCP cluster cluster-zstream-4-19-7g9hqt: failed starting cluster creation "cluster-zstream-4-19-7g9hqt" in resourcegroup="rg-zstream-upgrade-4-19-ld7rf2": PUT https://rp.j4591744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-ld7rf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-7g9hqt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-7g9hqt: failed starting cluster creation \"cluster-zstream-4-19-7g9hqt\" in resourcegroup=\"rg-zstream-upgrade-4-19-ld7rf2\": PUT https://rp.j4591744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-ld7rf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-7g9hqt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007e2660>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-7g9hqt\" in resourcegroup=\"rg-zstream-upgrade-4-19-ld7rf2\": PUT https://rp.j4591744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-ld7rf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-7g9hqt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0013af0b0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["39"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "1afb2267-84f2-4151-be67-4ec123738dbb",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:36:51 GMT",
                        ],
                        "Content-Length": ["138"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0007e25a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0007e2680>: 
    failed to create HCP cluster cluster-zstream-4-19-7g9hqt: failed starting cluster creation "cluster-zstream-4-19-7g9hqt" in resourcegroup="rg-zstream-upgrade-4-19-ld7rf2": PUT https://rp.j4591744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-ld7rf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-7g9hqt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-7g9hqt: failed starting cluster creation \"cluster-zstream-4-19-7g9hqt\" in resourcegroup=\"rg-zstream-upgrade-4-19-ld7rf2\": PUT https://rp.j4591744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-ld7rf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-7g9hqt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007e2660>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-7g9hqt\" in resourcegroup=\"rg-zstream-upgrade-4-19-ld7rf2\": PUT https://rp.j4591744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-ld7rf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-7g9hqt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0013af0b0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["39"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "1afb2267-84f2-4151-be67-4ec123738dbb",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:36:51 GMT",
                        ],
                        "Content-Length": ["138"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0007e25a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 61fdb10 · merge 4fe7725
n/a e2e failed ★#4393 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
21:20:41 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000e590e0>: 
    failed to create HCP cluster cluster-zstream-4-19-l4l2ck: failed starting cluster creation "cluster-zstream-4-19-l4l2ck" in resourcegroup="rg-zstream-upgrade-4-19-t6x9rx": PUT https://rp.j1534080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-t6x9rx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-l4l2ck
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-l4l2ck: failed starting cluster creation \"cluster-zstream-4-19-l4l2ck\" in resourcegroup=\"rg-zstream-upgrade-4-19-t6x9rx\": PUT https://rp.j1534080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-t6x9rx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-l4l2ck\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e590c0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-l4l2ck\" in resourcegroup=\"rg-zstream-upgrade-4-19-t6x9rx\": PUT https://rp.j1534080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-t6x9rx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-l4l2ck\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00140a660>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["45"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "2ebdb90d-96fc-42d7-a5a3-1e6d181a2e73",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:37:04 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000e58fc0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000e590e0>: 
    failed to create HCP cluster cluster-zstream-4-19-l4l2ck: failed starting cluster creation "cluster-zstream-4-19-l4l2ck" in resourcegroup="rg-zstream-upgrade-4-19-t6x9rx": PUT https://rp.j1534080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-t6x9rx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-l4l2ck
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-l4l2ck: failed starting cluster creation \"cluster-zstream-4-19-l4l2ck\" in resourcegroup=\"rg-zstream-upgrade-4-19-t6x9rx\": PUT https://rp.j1534080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-t6x9rx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-l4l2ck\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e590c0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-l4l2ck\" in resourcegroup=\"rg-zstream-upgrade-4-19-t6x9rx\": PUT https://rp.j1534080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-t6x9rx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-l4l2ck\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00140a660>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["45"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "2ebdb90d-96fc-42d7-a5a3-1e6d181a2e73",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:37:04 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000e58fc0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 87a94b6 · merge e4ce686
n/a e2e failed ★#4539 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
20:56:00 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000a10f00>: 
    failed to create HCP cluster cluster-zstream-4-19-jvdvs7: failed starting cluster creation "cluster-zstream-4-19-jvdvs7" in resourcegroup="rg-zstream-upgrade-4-19-fmtc66": PUT https://rp.j2748160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-fmtc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jvdvs7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-jvdvs7: failed starting cluster creation \"cluster-zstream-4-19-jvdvs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-fmtc66\": PUT https://rp.j2748160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-fmtc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jvdvs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a10ee0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-jvdvs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-fmtc66\": PUT https://rp.j2748160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-fmtc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jvdvs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00137aab0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "30abdb1c-3027-441a-afe4-0c56dfefc480",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:03:52 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["55"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a10e20>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000a10f00>: 
    failed to create HCP cluster cluster-zstream-4-19-jvdvs7: failed starting cluster creation "cluster-zstream-4-19-jvdvs7" in resourcegroup="rg-zstream-upgrade-4-19-fmtc66": PUT https://rp.j2748160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-fmtc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jvdvs7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-jvdvs7: failed starting cluster creation \"cluster-zstream-4-19-jvdvs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-fmtc66\": PUT https://rp.j2748160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-fmtc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jvdvs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a10ee0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-jvdvs7\" in resourcegroup=\"rg-zstream-upgrade-4-19-fmtc66\": PUT https://rp.j2748160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-fmtc66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jvdvs7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00137aab0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "30abdb1c-3027-441a-afe4-0c56dfefc480",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 22:03:52 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["55"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a10e20>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 56bf314 · merge 56bf314
n/a e2e failed ★#4485 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
20:39:18 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000759300>: 
    failed to create HCP cluster cluster-zstream-4-19-zvpj7f: failed starting cluster creation "cluster-zstream-4-19-zvpj7f" in resourcegroup="rg-zstream-upgrade-4-19-nrk9nl": PUT https://rp.j9657088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nrk9nl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zvpj7f
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-zvpj7f: failed starting cluster creation \"cluster-zstream-4-19-zvpj7f\" in resourcegroup=\"rg-zstream-upgrade-4-19-nrk9nl\": PUT https://rp.j9657088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nrk9nl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zvpj7f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007592e0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-zvpj7f\" in resourcegroup=\"rg-zstream-upgrade-4-19-nrk9nl\": PUT https://rp.j9657088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nrk9nl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zvpj7f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000b6ec60>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["43"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "08392dbf-202a-47fa-8b39-2f441a2a76f8",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:44:37 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0007591a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000759300>: 
    failed to create HCP cluster cluster-zstream-4-19-zvpj7f: failed starting cluster creation "cluster-zstream-4-19-zvpj7f" in resourcegroup="rg-zstream-upgrade-4-19-nrk9nl": PUT https://rp.j9657088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nrk9nl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zvpj7f
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-zvpj7f: failed starting cluster creation \"cluster-zstream-4-19-zvpj7f\" in resourcegroup=\"rg-zstream-upgrade-4-19-nrk9nl\": PUT https://rp.j9657088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nrk9nl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zvpj7f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007592e0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-zvpj7f\" in resourcegroup=\"rg-zstream-upgrade-4-19-nrk9nl\": PUT https://rp.j9657088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nrk9nl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-zvpj7f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000b6ec60>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["43"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "08392dbf-202a-47fa-8b39-2f441a2a76f8",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:44:37 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0007591a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 56bf314 · merge 56bf314
n/a e2e failed ★#4485 (merged) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
20:36:37 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0007d97e0>: 
    failed to create HCP cluster cluster-zstream-4-19-g8q84l: failed starting cluster creation "cluster-zstream-4-19-g8q84l" in resourcegroup="rg-zstream-upgrade-4-19-2s97kp": PUT https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-2s97kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g8q84l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-g8q84l: failed starting cluster creation \"cluster-zstream-4-19-g8q84l\" in resourcegroup=\"rg-zstream-upgrade-4-19-2s97kp\": PUT https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-2s97kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g8q84l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007d97c0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-g8q84l\" in resourcegroup=\"rg-zstream-upgrade-4-19-2s97kp\": PUT https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-2s97kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g8q84l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012f8fc0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "a749caaf-f103-4471-9de3-54f2e13526a1",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:48:10 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["69"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0007d96a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0007d97e0>: 
    failed to create HCP cluster cluster-zstream-4-19-g8q84l: failed starting cluster creation "cluster-zstream-4-19-g8q84l" in resourcegroup="rg-zstream-upgrade-4-19-2s97kp": PUT https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-2s97kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g8q84l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-g8q84l: failed starting cluster creation \"cluster-zstream-4-19-g8q84l\" in resourcegroup=\"rg-zstream-upgrade-4-19-2s97kp\": PUT https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-2s97kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g8q84l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007d97c0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-g8q84l\" in resourcegroup=\"rg-zstream-upgrade-4-19-2s97kp\": PUT https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-2s97kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-g8q84l\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012f8fc0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "a749caaf-f103-4471-9de3-54f2e13526a1",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:48:10 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["69"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0007d96a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4def80b · merge ec8c17a
n/a e2e failed ⚠#4505 (merged) 3
e2e (3)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
20:25:33 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:143]: Unexpected error:
    <*fmt.wrapErrors | 0xc00141aea0>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-vl8jff" in resourcegroup="rg-np-version-upgrade-vl8jff-6ld5tp" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0005c2e60>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp",
            },
            <*fmt.wrapError | 0xc000a9b0c0>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:143]: Unexpected error:
    <*fmt.wrapErrors | 0xc00141aea0>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-vl8jff" in resourcegroup="rg-np-version-upgrade-vl8jff-6ld5tp" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0005c2e60>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-vl8jff-6ld5tp",
            },
            <*fmt.wrapError | 0xc000a9b0c0>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-vl8jff\" in resourcegroup=\"rg-np-version-upgrade-vl8jff-6ld5tp\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
20:25:33 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000318400>: 
    failed to create HCP cluster cluster-zstream-4-19-kbdz2n: failed starting cluster creation "cluster-zstream-4-19-kbdz2n" in resourcegroup="rg-zstream-upgrade-4-19-j8vzrp": PUT https://rp.j7993856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-j8vzrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-kbdz2n
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-kbdz2n: failed starting cluster creation \"cluster-zstream-4-19-kbdz2n\" in resourcegroup=\"rg-zstream-upgrade-4-19-j8vzrp\": PUT https://rp.j7993856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-j8vzrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-kbdz2n\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0003183e0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-kbdz2n\" in resourcegroup=\"rg-zstream-upgrade-4-19-j8vzrp\": PUT https://rp.j7993856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-j8vzrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-kbdz2n\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000de2b10>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "4b295f70-546d-482a-95da-14e969a9d864",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:38:55 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["61"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000318320>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000318400>: 
    failed to create HCP cluster cluster-zstream-4-19-kbdz2n: failed starting cluster creation "cluster-zstream-4-19-kbdz2n" in resourcegroup="rg-zstream-upgrade-4-19-j8vzrp": PUT https://rp.j7993856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-j8vzrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-kbdz2n
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-kbdz2n: failed starting cluster creation \"cluster-zstream-4-19-kbdz2n\" in resourcegroup=\"rg-zstream-upgrade-4-19-j8vzrp\": PUT https://rp.j7993856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-j8vzrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-kbdz2n\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0003183e0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-kbdz2n\" in resourcegroup=\"rg-zstream-upgrade-4-19-j8vzrp\": PUT https://rp.j7993856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-j8vzrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-kbdz2n\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000de2b10>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "4b295f70-546d-482a-95da-14e969a9d864",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:38:55 GMT",
                        ],
                        "Content-Length": ["138"],
                        "X-Envoy-Upstream-Service-Time": ["61"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000318320>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
20:25:33 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:117]: Timed out after 2400.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ccd290>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial "4.21.5"
    {
        errs: [
            <*fmt.wrapError | 0xc00099e540>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                err: <*errors.errorString | 0xc0009b2510>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:117]: Timed out after 2400.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ccd290>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial "4.21.5"
    {
        errs: [
            <*fmt.wrapError | 0xc00099e540>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                err: <*errors.errorString | 0xc0009b2510>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                },
            },
        ],
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fa604f2
n/a e2e failed ★#4603 (open) 1
e2e (1)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
20:16:18 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0002f42e0>: 
    failed to create HCP cluster cluster-zstream-4-19-jp64d5: failed starting cluster creation "cluster-zstream-4-19-jp64d5" in resourcegroup="rg-zstream-upgrade-4-19-vmnpdh": PUT https://rp.j2595456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vmnpdh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jp64d5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-jp64d5: failed starting cluster creation \"cluster-zstream-4-19-jp64d5\" in resourcegroup=\"rg-zstream-upgrade-4-19-vmnpdh\": PUT https://rp.j2595456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vmnpdh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jp64d5\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0002f42c0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-jp64d5\" in resourcegroup=\"rg-zstream-upgrade-4-19-vmnpdh\": PUT https://rp.j2595456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vmnpdh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jp64d5\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000bc8540>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["94"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "ae348005-c559-47a1-a344-e4f5a595dd0e",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:18:09 GMT",
                        ],
                        "Content-Length": ["138"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0002f40c0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0002f42e0>: 
    failed to create HCP cluster cluster-zstream-4-19-jp64d5: failed starting cluster creation "cluster-zstream-4-19-jp64d5" in resourcegroup="rg-zstream-upgrade-4-19-vmnpdh": PUT https://rp.j2595456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vmnpdh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jp64d5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.19.3-candidate' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-jp64d5: failed starting cluster creation \"cluster-zstream-4-19-jp64d5\" in resourcegroup=\"rg-zstream-upgrade-4-19-vmnpdh\": PUT https://rp.j2595456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vmnpdh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jp64d5\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0002f42c0>{
            msg: "failed starting cluster creation \"cluster-zstream-4-19-jp64d5\" in resourcegroup=\"rg-zstream-upgrade-4-19-vmnpdh\": PUT https://rp.j2595456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-vmnpdh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jp64d5\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Version 'openshift-v4.19.3-candidate' doesn't exist\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000bc8540>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["94"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "ae348005-c559-47a1-a344-e4f5a595dd0e",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 21:18:09 GMT",
                        ],
                        "Content-Length": ["138"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0002f40c0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 86, 101, 114, 115, 105, 111, 110, 32, 39, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 118, 52, 46, 49, 57, 46, 51, 45, 99, 97, 110, 100, 105, 100, 97, 116, 101, 39, 32, 100, 111, 101, 115, 110, 39, 116, 32, 101, 120, 105, 115, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 138,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "PUT",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head afaa40f · merge 582667c
n/a n/a passed #4588 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 428d5ac
n/a e2e failed ⚠#4610 (open) 1
e2e (1)
cluster Properties.API.URL was nil
19:19:59 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: cluster Properties.API.URL was nil
Expected
    <*string | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: cluster Properties.API.URL was nil
Expected
    <*string | 0x0>: nil
not to be nil
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 85cd703
n/a n/a passed #4566 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head ac3354a · merge ac3354a
n/a n/a passed #4234 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4def80b · merge ec8c17a
n/a e2e failed ⚠#4505 (merged) 2
e2e (2)
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
18:17:10 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:117]: Timed out after 2400.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f3fec0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial "4.21.5"
    {
        errs: [
            <*fmt.wrapError | 0xc0004657a0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                err: <*errors.errorString | 0xc000a1e7d0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:117]: Timed out after 2400.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f3fec0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial "4.21.5"
    {
        errs: [
            <*fmt.wrapError | 0xc0004657a0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.5) failed: no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                err: <*errors.errorString | 0xc000a1e7d0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.5\"",
                },
            },
        ],
    }
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
18:17:10 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:143]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015a88d0>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-c9df8c-4rs6vp, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-c9df8c" in resourcegroup="rg-np-version-upgrade-c9df8c-4rs6vp" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-c9df8c-4rs6vp, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-c9df8c\" in resourcegroup=\"rg-np-version-upgrade-c9df8c-4rs6vp\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a896c0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-c9df8c-4rs6vp",
            },
            <*fmt.wrapError | 0xc0001706a0>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-c9df8c\" in resourcegroup=\"rg-np-version-upgrade-c9df8c-4rs6vp\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:143]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015a88d0>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-c9df8c-4rs6vp, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-c9df8c" in resourcegroup="rg-np-version-upgrade-c9df8c-4rs6vp" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-c9df8c-4rs6vp, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-c9df8c\" in resourcegroup=\"rg-np-version-upgrade-c9df8c-4rs6vp\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a896c0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-c9df8c-4rs6vp",
            },
            <*fmt.wrapError | 0xc0001706a0>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-c9df8c\" in resourcegroup=\"rg-np-version-upgrade-c9df8c-4rs6vp\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 60ec28b · merge f6ed2d9
n/a n/a passed #4599 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 56bf314 · merge 56bf314
n/a n/a passed #4485 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 5e2c098 · merge db17c8e
n/a n/a passed #4607 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4def80b · merge ec8c17a
n/a provision failed ⚠#4505 (merged) 1
provision (1)
Error: failed to resolve sha256:<hex>: GET "<url>": response status code 429: toomanyrequests: Quota exceeded for quota metric 'Requests per project per region' and limit 'Requests per project per region per minute per region' of service
16:52:15 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/global/mirror-image · suite step graph
Show raw failure
time=2026-03-24T17:34:18.969Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-24T17:34:29.061Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build05.ci.openshift.org registry.build05.ci.openshift.org registry.build05.ci.openshift.org registry.build05.ci.openshift.org\nFetch pull secret for source registry registry.k8s.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 to arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5.\nThe image will still be available under it's original digest sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 in the target registry.\nError: failed to resolve sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5: GET \"https://us-central1-docker.pkg.dev/v2/k8s-artifacts-prod/images/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": response status code 429: toomanyrequests: Quota exceeded for quota metric 'Requests per project per region' and limit 'Requests per project per region per minute per region' of service 'artifactregistry.googleapis.com' for consumer 'project_number:388270116193'.\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 87a94b6 · merge e4ce686
n/a e2e failed #4539 (merged) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
16:40:14 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015a8f00>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-wf2tbz" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-wf2tbz, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-wf2tbz\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-wf2tbz, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00063e230>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-wf2tbz",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015a8f00>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-wf2tbz" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-wf2tbz, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-wf2tbz\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-wf2tbz, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00063e230>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-wf2tbz",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 87a94b6 · merge e4ce686
n/a n/a passed #4539 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 87a94b6 · merge e4ce686
n/a n/a passed #4539 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9e233cc
n/a n/a passed #4556 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 98f7418 · merge 445d308
n/a n/a passed #3973 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head bd2f0e6 · merge bd2f0e6
n/a n/a passed #4597 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head ff7df37 · merge f9b0dd4
n/a provision failed #4257 (merged) 1
provision (1)
resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress context deadline exceeded
16:09:55 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Frontend/service/deploy · suite step graph
Show raw failure
time=2026-03-24T17:29:21.736Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-24T17:29:22.375Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy values="map[adminApi:map[namespace:aro-hcp-admin-api serviceAccount:admin-api] adminApiHost:admin.j8504704.hcpsvc.osadev.cloud apiBindPort:8443 audit:map[connectSocket:false] azure:map[clientId:6f21f871-04b5-4824-8b7a-450cf197cf47 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[databaseName:arohcpprow-rp-j8504704 databaseUrl:https://arohcpprow-rp-j8504704.documents.azure.com:443/ frontendMiClientId:7b104339-222d-48cb-a095-d5a2d2b28c21 location:westus3] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:frontend-cert-prow-j8504704] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-zzwvt2zp/pipeline@sha256:f136210a840ccb8ad45d2c57a2f5d3997d2770bcda339b150295813991463c3d replicas:2 zoneCount:4] exitOnPanic:true mise:map[deploy:false policyLabel:ARM Policy] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-zzwvt2zp/pipeline:pull workloadIdentityClientId:e917b0cb-bbfd-4444-ae15-0d5594f67823 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:frontend workloadIdentityClientId:7b104339-222d-48cb-a095-d5a2d2b28c21 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp] virtualService:map[host:rp.j8504704.hcpsvc.osadev.cloud]]"
time=2026-03-24T17:29:22.375Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy
time=2026-03-24T17:29:22.444Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-24T17:29:22.582Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy dryRun=false
time=2026-03-24T17:29:22.582Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy
time=2026-03-24T17:29:22.715Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy
time=2026-03-24T17:34:27.015Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-24T17:34:27.015Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy
time=2026-03-24T17:34:27.015Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy
time=2026-03-24T17:34:31.048Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy release=aro-hcp-frontend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-frontend-dev\" failed: resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[adminApi:map[namespace:aro-hcp-admin-api serviceAccount:admin-api] adminApiHost:admin.j8504704.hcpsvc.osadev.cloud apiBindPort:8443 audit:map[connectSocket:false] azure:map[clientId:6f21f871-04b5-4824-8b7a-450cf197cf47 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[databaseName:arohcpprow-rp-j8504704 databaseUrl:https://arohcpprow-rp-j8504704.documents.azure.com:443/ frontendMiClientId:7b104339-222d-48cb-a095-d5a2d2b28c21 location:westus3] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:frontend-cert-prow-j8504704] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-zzwvt2zp/pipeline@sha256:f136210a840ccb8ad45d2c57a2f5d3997d2770bcda339b150295813991463c3d replicas:2 zoneCount:4] exitOnPanic:true mise:map[deploy:false policyLabel:ARM Policy] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-zzwvt2zp/pipeline:pull workloadIdentityClientId:e917b0cb-bbfd-4444-ae15-0d5594f67823 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:frontend workloadIdentityClientId:7b104339-222d-48cb-a095-d5a2d2b28c21 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp] virtualService:map[host:rp.j8504704.hcpsvc.osadev.cloud]]"
time=2026-03-24T17:34:31.051Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5xUTW8TQQy9768Y7amRdqqyragI6qEfiAMqigTiUMRhdsbZuJ21Vx5vQlF/PNqmIduQhtDrs9/zs8fjLIIagcSdeEjmzASnTl0V4YCrW/D6CSmMkwpSXZgl9Nk18AeCOdAjklrnV/Doe2aMMfkl0xTra9fmhcmnwqRAwfpHtIecsJ35Ni+W6R+dwsLdDyK2XkIWfioIufgYu0sWkyJbpFogpRX/C3gBnQjPMYBcRteHBoWTb//Blzl6uGZCZRnaWElsMf0NRTsXn7hD0jzt4k04XGGSrlVkuuhCDbpnxadS595zRzrs8OXk/5Pelm0bUEGftrCuoI183wDt28FzwnoxQBSn6J2CFZgKpBnIzhc79zLpYrxACkj1M7G2i9FW68CGhfNOZyz4y/Xjn3BEv9y7GHlhXWiQrGvRKu/qY6fI07xezSfW2Su8b8zp770DkF4ASPtJI9O+r539eJ/ddRUIgUL60H/8lD2YxQwEjGIDSV3Tmgp0AUDmIDiFHj7Iy6PyrT06tuXJ1zen4/LduCxv8pE5PDQv5RyfjMvTm3w0yh7MLSOZO6RwhkQgg3PFZNZXanieNu9S9mBa4T66NlqYlsMyeT+RwkSuC9NASq7uJVkCiKnuB80HSP53AAAA//8soq4JVAUAAA=="
time=2026-03-24T17:34:31.055Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy
time=2026-03-24T17:34:31.055Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-frontend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0b221bd
n/a unknown failed #4477 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 953df9e · merge 953df9e
n/a n/a passed #4605 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 67bf3e2 · merge 67bf3e2
n/a e2e failed #4596 (merged) 1
e2e (1)
route host was never found: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
15:37:36 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
    <*fmt.wrapError | 0xc00076a140>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud on 172.30.0.10:53: no such host
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud on 172.30.0.10:53: no such host",
        err: <*url.Error | 0xc0009f6a50>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000e343c0>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: nil,
                Err: <*net.DNSError | 0xc00081a500>{
                    UnwrapErr: nil,
                    Err: "no such host",
                    Name: "agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud",
                    Server: "172.30.0.10:53",
                    IsTimeout: false,
                    IsTemporary: false,
                    IsNotFound: true,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
    <*fmt.wrapError | 0xc00076a140>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud on 172.30.0.10:53: no such host
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud on 172.30.0.10:53: no such host",
        err: <*url.Error | 0xc0009f6a50>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000e343c0>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: nil,
                Err: <*net.DNSError | 0xc00081a500>{
                    UnwrapErr: nil,
                    Err: "no such host",
                    Name: "agnhost-e2e-serving-app-p5wcz.apps.aro.w2u4i3e8k2c2x4k.aukg.j8206464.hcp.osadev.cloud",
                    Server: "172.30.0.10:53",
                    IsTimeout: false,
                    IsTemporary: false,
                    IsNotFound: true,
                },
            },
        },
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ac3354a · merge ac3354a
n/a n/a passed #4234 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 7cddbd3 · merge 4fe7725
n/a n/a passed #4393 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head faaaa2b
n/a e2e failed #4513 (open) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
15:26:44 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc00087b2f0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-vs482z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vs482z, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-vs482z\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vs482z, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f77b50>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vs482z",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc00087b2f0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-vs482z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vs482z, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-vs482z\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vs482z, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f77b50>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vs482z",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head d08cdff · merge 453429f
n/a n/a passed #4582 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a053b39 · merge bd2f0e6
n/a n/a passed #4597 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head fffb2dd · merge a1dbbe7
n/a n/a passed #4604 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c49a0 · merge a3c49a0
n/a e2e failed #4510 (merged) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
14:51:11 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc000513230>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-qk9ftl" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qk9ftl, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-qk9ftl\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qk9ftl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0002f9b00>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qk9ftl",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc000513230>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-qk9ftl" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qk9ftl, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-qk9ftl\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qk9ftl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0002f9b00>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qk9ftl",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head a3c49a0 · merge a3c49a0
n/a e2e failed #4510 (merged) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
14:48:19 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc0005ffbf0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-bwmxx9" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bwmxx9, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-bwmxx9\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bwmxx9, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a05670>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bwmxx9",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc0005ffbf0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-bwmxx9" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bwmxx9, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-bwmxx9\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bwmxx9, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a05670>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bwmxx9",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 047f439
n/a e2e failed #4556 (open) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
14:45:25 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc000972d50>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-mxd8xb" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mxd8xb, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-mxd8xb\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mxd8xb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001643310>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mxd8xb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc000972d50>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-mxd8xb" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mxd8xb, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-mxd8xb\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mxd8xb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001643310>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mxd8xb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4a35705 · merge 4a35705
n/a n/a passed #4602 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9695199
n/a n/a passed #4603 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 6b3bb40 · merge 0e19405
n/a n/a passed #4507 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head cab9df4 · merge cab9df4
n/a n/a passed #4554 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 87a94b6 · merge e4ce686
n/a n/a passed #4539 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head eec30d3 · merge d4656bc
n/a provision failed #4601 (merged) 1
provision (1)
resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress context deadline exceeded
13:32:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/aro-hcp-exporter · suite step graph
Show raw failure
time=2026-03-24T14:08:44.893Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter description="Step aro-hcp-exporter\n Kind: Helm\n"
time=2026-03-24T14:08:45.577Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter values="map[cacheTTL:60m clusterNames:prow-j9260032-svc,prow-j9260032-mgmt-1 collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:service-tag-usage,kusto-logs-current image:map[digest:sha256:38294123323d6fd6b64bf8095ad5cb1d11997023e20d61349d47f6615318205e registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] kustRegion:East US 2 kustoCluster:hcp-dev-us-2 listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:c43973ee-69d4-4271-a612-6159f430ce81 name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-24T14:08:45.577Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-24T14:08:45.762Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter namespace=aro-hcp-exporter
time=2026-03-24T14:08:46.066Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter dryRun=false
time=2026-03-24T14:08:46.066Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-24T14:08:46.187Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-24T14:13:49.471Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-24T14:13:49.471Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-24T14:13:49.471Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-24T14:13:51.743Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter release=aro-hcp-exporter namespace=aro-hcp-exporter status=failed description="Release \"aro-hcp-exporter\" failed: resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[cacheTTL:60m clusterNames:prow-j9260032-svc,prow-j9260032-mgmt-1 collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:service-tag-usage,kusto-logs-current image:map[digest:sha256:38294123323d6fd6b64bf8095ad5cb1d11997023e20d61349d47f6615318205e registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] kustRegion:East US 2 kustoCluster:hcp-dev-us-2 listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:c43973ee-69d4-4271-a612-6159f430ce81 name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-24T14:13:51.744Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yRT0sDMRDF7/sphj11Ybf0z1J0pQdBT6IXPVU8ZJNHm3Y3E5Jpa6EfXtairULBev1l3ksyv6SBUEDkddCINCWjRImqG/S4XkLLg3WmihKsm+d0QE+qxTfCBu6TRK/0F85eEyKi9BlhYzUe2VnhkOaUqsDFQvsC756D4Az7kb7VmtdO/pm+NHYH3/CuxQUXJm83yWpdIzgI4n23kJjsabtAAIltEUW1nmrIFnDUM0rQ4V46GowmxWBcjMqXYVkNrqpyMksz6vfp3MxwXJXXszTLkj0t2TpaWWem1jmEE43s6GjvVNtvX8mefODu9PjQnDybw/DfSnJqeJ5TixjVvKvkYBCo3p183iDqjwAAAP//XMOyfGwCAAA="
time=2026-03-24T14:13:51.746Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-24T14:13:51.746Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 74cb42a · merge bd2f0e6
n/a n/a passed #4597 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 96e81da · merge ec8c17a
n/a n/a passed #4505 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7de281f · merge 7de281f
n/a n/a passed #4600 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 4eda003 · merge 4eda003
n/a n/a passed #4576 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4bcc7ca · merge 4bcc7ca
n/a n/a passed #4592 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c49a0 · merge a3c49a0
n/a n/a passed #4510 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 047f439
n/a n/a passed #4556 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 047f439
n/a n/a passed #4556 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 7bdf0d9 · merge f6ed2d9
n/a n/a passed #4599 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 724c3cb · merge 724c3cb
n/a n/a passed #4598 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head cab9df4 · merge cab9df4
n/a e2e failed #4554 (merged) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
11:37:22 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f832c0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-lpk849" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-lpk849, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-lpk849\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-lpk849, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0006279c0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-lpk849",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f832c0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-lpk849" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-lpk849, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-lpk849\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-lpk849, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0006279c0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-lpk849",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 87a94b6 · merge e4ce686
n/a e2e failed #4539 (merged) 1
e2e (1)
Interrupted by User
11:25:32 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 87a94b6 · merge e4ce686
n/a n/a passed #4539 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head ef72c11 · merge bd2f0e6
n/a unknown failed #4597 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 67bf3e2 · merge 67bf3e2
n/a n/a passed #4596 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 9a6bf6f · merge c367d34
n/a e2e failed #4595 (merged) 25
e2e (25)
Interrupted by User
09:53:55 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc001109140>: 
    failed to create HCP cluster ea-listnsdk6p, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: failed waiting for cluster="ea-listnsdk6p" in resourcegroup="ea-list-2qfkbm" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listnsdk6p, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: failed waiting for cluster=\"ea-listnsdk6p\" in resourcegroup=\"ea-list-2qfkbm\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010ea6a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm",
            },
            <*fmt.wrapErrors | 0xc001109110>{
                msg: "failed waiting for cluster=\"ea-listnsdk6p\" in resourcegroup=\"ea-list-2qfkbm\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010ea6a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc001109140>: 
    failed to create HCP cluster ea-listnsdk6p, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: failed waiting for cluster="ea-listnsdk6p" in resourcegroup="ea-list-2qfkbm" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listnsdk6p, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: failed waiting for cluster=\"ea-listnsdk6p\" in resourcegroup=\"ea-list-2qfkbm\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010ea6a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm",
            },
            <*fmt.wrapErrors | 0xc001109110>{
                msg: "failed waiting for cluster=\"ea-listnsdk6p\" in resourcegroup=\"ea-list-2qfkbm\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010ea6a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listnsdk6p in resource group ea-list-2qfkbm",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f11980>: 
    failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-gbm28q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-gbm28q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010d12a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q",
            },
            <*fmt.wrapErrors | 0xc000f11950>{
                msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-gbm28q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010d12a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f11980>: 
    failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-gbm28q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-gbm28q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010d12a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q",
            },
            <*fmt.wrapErrors | 0xc000f11950>{
                msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-gbm28q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010d12a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-gbm28q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapErrors | 0xc0005dab40>: 
    failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-55std9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-55std9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015d0c20>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9",
            },
            <*fmt.wrapErrors | 0xc0005dab10>{
                msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-55std9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0015d0c20>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapErrors | 0xc0005dab40>: 
    failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-55std9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-55std9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015d0c20>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9",
            },
            <*fmt.wrapErrors | 0xc0005dab10>{
                msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-55std9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0015d0c20>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-55std9",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
09:53:55 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d82ab0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zmbms2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zmbms2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000dc1d40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2",
            },
            <*fmt.wrapErrors | 0xc000d82a80>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zmbms2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000dc1d40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d82ab0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zmbms2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zmbms2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000dc1d40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2",
            },
            <*fmt.wrapErrors | 0xc000d82a80>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zmbms2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000dc1d40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zmbms2",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
09:53:55 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:121]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:121]: Interrupted by User
Interrupted by User
09:53:55 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
Interrupted by User
09:53:55 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:120]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:120]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc0007a9170>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-ngfjrh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ngfjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000dd6600>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh",
            },
            <*fmt.wrapErrors | 0xc0007a9050>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ngfjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000dd6600>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc0007a9170>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-ngfjrh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ngfjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000dd6600>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh",
            },
            <*fmt.wrapErrors | 0xc0007a9050>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ngfjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000dd6600>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-ngfjrh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011f4ff0>: 
    failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-pv6phf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-pv6phf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011dab40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf",
            },
            <*fmt.wrapErrors | 0xc0011f4fc0>{
                msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-pv6phf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011dab40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011f4ff0>: 
    failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-pv6phf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-pv6phf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011dab40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf",
            },
            <*fmt.wrapErrors | 0xc0011f4fc0>{
                msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-pv6phf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011dab40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-pv6phf",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc00109efc0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-8wnpmk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-8wnpmk, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-8wnpmk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-8wnpmk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fcc9b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-8wnpmk",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc00109efc0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-8wnpmk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-8wnpmk, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-8wnpmk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-8wnpmk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fcc9b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-8wnpmk",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc0006b6f30>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2s59zq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s59zq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fddd80>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq",
            },
            <*fmt.wrapErrors | 0xc0006b6f00>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s59zq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fddd80>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc0006b6f30>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2s59zq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s59zq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fddd80>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq",
            },
            <*fmt.wrapErrors | 0xc0006b6f00>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s59zq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fddd80>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-2s59zq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc000eb7200>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-slq2bs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-slq2bs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010bf640>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs",
            },
            <*fmt.wrapErrors | 0xc000eb71d0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-slq2bs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010bf640>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc000eb7200>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-slq2bs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-slq2bs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010bf640>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs",
            },
            <*fmt.wrapErrors | 0xc000eb71d0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-slq2bs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010bf640>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-slq2bs",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e7e9f0>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-blsl7w" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-blsl7w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000d9d940>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w",
            },
            <*fmt.wrapErrors | 0xc000e7e9c0>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-blsl7w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000d9d940>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e7e9f0>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-blsl7w" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-blsl7w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000d9d940>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w",
            },
            <*fmt.wrapErrors | 0xc000e7e9c0>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-blsl7w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000d9d940>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-blsl7w",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc001129500>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-msj7rl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-msj7rl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e85520>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl",
            },
            <*fmt.wrapErrors | 0xc0011294d0>{
                msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-msj7rl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e85520>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc001129500>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-msj7rl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-msj7rl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e85520>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl",
            },
            <*fmt.wrapErrors | 0xc0011294d0>{
                msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-msj7rl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e85520>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-msj7rl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
09:53:55 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc001403410>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-bd54jc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bd54jc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00101f780>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc",
            },
            <*fmt.wrapErrors | 0xc0014033e0>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bd54jc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00101f780>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc001403410>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-bd54jc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bd54jc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00101f780>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc",
            },
            <*fmt.wrapErrors | 0xc0014033e0>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bd54jc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00101f780>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bd54jc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009568a0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-52t5m6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-52t5m6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fd7090>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6",
            },
            <*fmt.wrapErrors | 0xc000956870>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-52t5m6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fd7090>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009568a0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-52t5m6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-52t5m6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fd7090>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6",
            },
            <*fmt.wrapErrors | 0xc000956870>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-52t5m6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fd7090>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-52t5m6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
09:53:55 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e0a8d0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-5qzmjb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-5qzmjb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fc3150>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb",
            },
            <*fmt.wrapErrors | 0xc000e0a8a0>{
                msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-5qzmjb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fc3150>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e0a8d0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-5qzmjb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-5qzmjb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fc3150>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb",
            },
            <*fmt.wrapErrors | 0xc000e0a8a0>{
                msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-5qzmjb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fc3150>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-5qzmjb",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014ce9c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-s4lrjc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-s4lrjc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011c0190>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc",
            },
            <*fmt.wrapErrors | 0xc0014ce990>{
                msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-s4lrjc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011c0190>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014ce9c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-s4lrjc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-s4lrjc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011c0190>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc",
            },
            <*fmt.wrapErrors | 0xc0014ce990>{
                msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-s4lrjc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011c0190>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-s4lrjc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc0007feea0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4cwxb8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-4cwxb8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00126cb10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8",
            },
            <*fmt.wrapErrors | 0xc0007fee70>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-4cwxb8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00126cb10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc0007feea0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4cwxb8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-4cwxb8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00126cb10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8",
            },
            <*fmt.wrapErrors | 0xc0007fee70>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-4cwxb8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00126cb10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-4cwxb8",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:53:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc00092f350>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-v9djvs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-v9djvs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001179bf0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs",
            },
            <*fmt.wrapErrors | 0xc00092f320>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-v9djvs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001179bf0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc00092f350>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-v9djvs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-v9djvs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001179bf0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs",
            },
            <*fmt.wrapErrors | 0xc00092f320>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-v9djvs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001179bf0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-v9djvs",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Cluster should become ready within 45 minutes
09:53:55 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 9de7f0c · merge 8b7576c
n/a e2e failed #4473 (merged) 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:51:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapErrors | 0xc00124ee10>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-dp5nl9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-dp5nl9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011be660>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9",
            },
            <*fmt.wrapErrors | 0xc00124ede0>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-dp5nl9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011be660>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapErrors | 0xc00124ee10>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-dp5nl9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-dp5nl9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011be660>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9",
            },
            <*fmt.wrapErrors | 0xc00124ede0>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-dp5nl9\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011be660>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-dp5nl9",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:51:08 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e7c7e0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-mx72rw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mx72rw, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-mx72rw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mx72rw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013c2170>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mx72rw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e7c7e0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-mx72rw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mx72rw, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-mx72rw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mx72rw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013c2170>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mx72rw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 8ac2a50 · merge 8ac2a50
n/a n/a passed #4442 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 39d0065 · merge 39d0065
n/a n/a passed #4583 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 80b2e3c · merge 80b2e3c
n/a n/a passed #4594 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 1ba8557 · merge cab9df4
n/a n/a passed #4554 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 047f439
n/a n/a passed #4556 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 162efb4 · merge 162efb4
n/a n/a passed #4593 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 25d2db7 · merge 4bcc7ca
n/a n/a passed #4592 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5ed7b32 · merge 5ed7b32
n/a n/a passed #4585 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 5f59c6e · merge 39d0065
n/a provision failed ⚠#4583 (merged) 1
provision (1)
failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name="fluentbit"].limits: field not declared in schema
07:36:54 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/arobit · suite step graph
Show raw failure
time=2026-03-24T08:14:37.126Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-03-24T08:14:37.623Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:prow-j7437568-mgmt-1 clusterType:mgmt dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:33bed43b09e3d2384c7661c74f6ed79ca53b0c25a3e28d77f139a7b9d50fbf1a pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit]] geoShortId:us kusto:map[enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net serviceLogsDatabase:ServiceLogs] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan: clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-prow-j7437568-mgmt-1 rpAccountName:placeholder rpNamespace: rpSan: rpSecretName:rplogs] image:map[digest:sha256:2ae5282c99fa8c82bb394b23dbcb9ea0fa553e4ec3cce9e39d110f44b6282062 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e5c0d020-14b6-44fa-9911-d456e0602838 region:westus3 secretProvider:map[keyVault:ah-prow-mg-j7437568-1] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-24T08:14:37.623Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit
time=2026-03-24T08:14:37.957Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit namespace=arobit
time=2026-03-24T08:14:38.239Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit dryRun=false
time=2026-03-24T08:14:38.239Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit
time=2026-03-24T08:14:38.370Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit
time=2026-03-24T08:14:42.079Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit err="failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name=\"fluentbit\"].limits: field not declared in schema"
time=2026-03-24T08:14:42.079Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit
time=2026-03-24T08:14:42.079Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit
time=2026-03-24T08:14:42.703Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit release=arobit namespace=arobit status=failed description="Release \"arobit\" failed: failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name=\"fluentbit\"].limits: field not declared in schema" values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:prow-j7437568-mgmt-1 clusterType:mgmt dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:33bed43b09e3d2384c7661c74f6ed79ca53b0c25a3e28d77f139a7b9d50fbf1a pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit]] geoShortId:us kusto:map[enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net serviceLogsDatabase:ServiceLogs] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan: clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-prow-j7437568-mgmt-1 rpAccountName:placeholder rpNamespace: rpSan: rpSecretName:rplogs] image:map[digest:sha256:2ae5282c99fa8c82bb394b23dbcb9ea0fa553e4ec3cce9e39d110f44b6282062 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e5c0d020-14b6-44fa-9911-d456e0602838 region:westus3 secretProvider:map[keyVault:ah-prow-mg-j7437568-1] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-24T08:14:42.703Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yRQU/qQBDH7/0Ukz3RpBBeIS+EFw5P48ngQT1hPGx3h7rQzjSzU4gJH95UUIhR4frb338yO/+kQgXByK04jDADb9WqLSrscbFCp7eB/DSqBCoz2KM7W+Mnwg3SO4mNdR84fUoAAMwDyiY4/O8ct6QmA2OFi6D9JcvWikc5MpPtM9dMy1DObXOhXrVRUe65wp8C36hXgXyg8lzisP9lmxzkOVNQljOZ5Plfsm4LFELFeNMdMSY72L6gIGioMaqtGyhQt4gEPW8VO9wz+TD/2x+O+vn4cTiZ/hlPR5OFSWEwgF+ccb4waZrsYMWBYB3IzwIRykn1THBs/LTqrx0nO2iEu9fjohk07PfyZUMyqLjMoMYYbdmN5O5AULyefN5jdG8BAAD//yumHLugAgAA"
time=2026-03-24T08:14:42.703Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit
time=2026-03-24T08:14:42.703Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to create typed patch object (arobit/arobit-forwarder; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name=\"fluentbit\"].limits: field not declared in schema"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ac3354a · merge ac3354a
n/a provision failed #4234 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code BadRequest; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/admin-api' was not found.; provider Microsoft.ManagedIdentity
07:36:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph
Show raw failure
time=2026-03-24T07:57:26.824Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-24T07:57:29.113Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-24T07:57:30.737Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8bffa6a59c30c76b6c0f52143355fc7794894833c46ae7842be2f2600ca890c4
time=2026-03-24T08:09:03.105Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9601280-svc/providers/Microsoft.Resources/deployments/8bffa6a59c30c76b6c0f52143355fc7794894833c46ae7842be2f2600ca890c4/operationStatuses/08584272678355634411\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9601280/providers/Microsoft.Resources/deployments/rp_cosmos_db\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"FailedIdentityOperation\\\",\\r\\n            \\\"message\\\": \\\"Identity operation for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9601280/providers/Microsoft.DocumentDB/databaseAccounts/arohcpprow-rp-j9601280' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\\\\\\\"error\\\\\\\":{\\\\\\\"code\\\\\\\":\\\\\\\"BadRequest\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9601280-svc/providers/Microsoft.ManagedIdentity/userAssignedIdentities/admin-api' was not found.\\\\\\\"}}'.'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 89b4c84 · merge 8b7576c
n/a e2e failed #4473 (merged) 3
e2e (3)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
07:34:06 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc00141a4e0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-pxhs5d" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-pxhs5d, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-pxhs5d\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-pxhs5d, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00086bb00>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-pxhs5d",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc00141a4e0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-pxhs5d" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-pxhs5d, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-pxhs5d\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-pxhs5d, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00086bb00>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-pxhs5d",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
07:34:06 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapError | 0xc0009b5440>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-5fwbrc": PUT https://rp.j1197056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fwbrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fwbrc\": PUT https://rp.j1197056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fwbrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009b5420>{
            msg: "failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fwbrc\": PUT https://rp.j1197056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fwbrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001511440>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["206"],
                        "X-Envoy-Upstream-Service-Time": ["40"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "502f5de0-2332-46f7-b52d-c5c3ef55b417",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 08:19:01 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0009b4600>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 206,
                    TransferEncoding: nil,
                ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapError | 0xc0009b5440>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-5fwbrc": PUT https://rp.j1197056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fwbrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fwbrc\": PUT https://rp.j1197056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fwbrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009b5420>{
            msg: "failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-5fwbrc\": PUT https://rp.j1197056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-5fwbrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001511440>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["206"],
                        "X-Envoy-Upstream-Service-Time": ["40"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "502f5de0-2332-46f7-b52d-c5c3ef55b417",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 08:19:01 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc0009b4600>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 206,
                    TransferEncoding: nil,
                ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
07:34:06 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ea49f0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-s94fkv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-s94fkv, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-s94fkv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-s94fkv, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0017d8870>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-s94fkv",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ea49f0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-s94fkv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-s94fkv, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-s94fkv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-s94fkv, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0017d8870>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-s94fkv",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4eda003 · merge 4eda003
n/a n/a passed #4576 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 25d2db7 · merge 4bcc7ca
n/a provision failed #4592 (merged) 1
provision (1)
resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded
06:43:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy · suite step graph
Show raw failure
time=2026-03-24T07:20:31.972Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-24T07:20:32.576Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj4483840.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj4483840.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:8f3a4d06-1d40-40ff-aca0-cb33dc4c01f1 databaseName:arohcpprow-rp-j4483840 databaseUrl:https://arohcpprow-rp-j4483840.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-kr5iz86v/pipeline@sha256:bac45a637794cb0862fcf1c22b4d7b6260af4f04cf3ed83cfc787e674f96d8fd replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-kr5iz86v/pipeline:pull workloadIdentityClientId:34f2105a-cd04-4a7f-8c10-cbab7c9fdabf workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:8f3a4d06-1d40-40ff-aca0-cb33dc4c01f1 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-24T07:20:32.576Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-24T07:20:32.848Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-24T07:20:33.195Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-03-24T07:20:33.195Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-24T07:20:33.345Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-24T07:25:42.434Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-24T07:25:42.434Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-24T07:25:42.434Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-24T07:25:46.647Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj4483840.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj4483840.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:8f3a4d06-1d40-40ff-aca0-cb33dc4c01f1 databaseName:arohcpprow-rp-j4483840 databaseUrl:https://arohcpprow-rp-j4483840.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-kr5iz86v/pipeline@sha256:bac45a637794cb0862fcf1c22b4d7b6260af4f04cf3ed83cfc787e674f96d8fd replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-kr5iz86v/pipeline:pull workloadIdentityClientId:34f2105a-cd04-4a7f-8c10-cbab7c9fdabf workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:8f3a4d06-1d40-40ff-aca0-cb33dc4c01f1 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-24T07:25:46.648Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTQW/aQBCF7/4VI59A8kYI0laiyoGmPVWpUNNTqh7Wu6+wYb1jza5BRPz4yrgEKzIR17fvm30zmsk8EgkiN2IQ6Y6sTjrp0mPE5TNM+u6CncckLqwK6qQfusKrhC3CUYm1Nid5/DsjIsoXRpaN919csC6s8oLyUpsNglV1470qz7oWVmtT50UH/mSPAcpDW4iCh0mOwwD4CNk6g96LOsEVkjgTB6CvqD3vK4Q0wA34F01as7gX3YZYsndmf3R5z7vTN+/gS0DaEgjJGf2mkSviPsII0lJ46yzk3usY+0OK3QjUBnu11Y1Pl8f0wMElluu6/s8sjOGmm9Rl7z2Hv271oOt+MP3SCJQ0IbkKyhwt17IX3e2mtPI7q5H9+ZxtmhISkBC/tQsbswPt1hBQmyUmXdVUIu2AQCOrE1p5lE8n049qMlPT21+TT/PpZD6bPeVjurmhi54P89vpUz4eZwd6Zhdo44K9cyFAemfGgc7X1T+rt/eUHagWbl/PQQuq2Xbm64oU5HlVUIUY9aotyWIhVO57zVtE8y8AAP//0PWTcQwEAAA="
time=2026-03-24T07:25:46.654Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-24T07:25:46.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 162cc07 · merge 162cc07
n/a n/a passed #4574 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 49f81d9 · merge 49f81d9
n/a n/a passed #4587 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 288e1d5
n/a provision failed ⚠#4589 (closed) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code RoleAssignmentExists; detail message The role assignment already exists. The ID of the existing role assignment is <hex>.
02:29:00 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/svc-mgmt-permissions · suite step graph
Show raw failure
time=2026-03-24T03:04:07.422Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions description="Step svc-mgmt-permissions\n  Kind: ARM\n  Template: templates/svc-mgmt-permissions.bicep\n  Parameters: configurations/svc-mgmt-permissions.tmpl.bicepparam"
time=2026-03-24T03:04:07.900Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions
time=2026-03-24T03:04:08.871Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions deployment=6f5dc51387bf3d351f1b8e5546f891a1318b5147b96d6ce634f7f9b9588dd09a
time=2026-03-24T03:04:39.110Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=svc-mgmt-permissions err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1837696-mgmt-1/providers/Microsoft.Resources/deployments/6f5dc51387bf3d351f1b8e5546f891a1318b5147b96d6ce634f7f9b9588dd09a/operationStatuses/08584272854372039071\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"RoleAssignmentExists\\\",\\r\\n    \\\"message\\\": \\\"The role assignment already exists. The ID of the existing role assignment is 0890b582518a5466b9bb14be0e9debd7.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c86c5ea · merge 8b7576c
n/a e2e failed #4473 (merged) 2
e2e (2)
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
01:41:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapError | 0xc000e07c20>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-ftgghp": PUT https://rp.j4076032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ftgghp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-ftgghp\": PUT https://rp.j4076032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ftgghp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e07c00>{
            msg: "failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-ftgghp\": PUT https://rp.j4076032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ftgghp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011a7080>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["206"],
                        "X-Envoy-Upstream-Service-Time": ["34"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "2542e3fe-ba0f-458c-8e87-13393388e6f3",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 02:30:15 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000e07b20>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 206,
                    TransferEncoding: nil,
                ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:122]: Unexpected error:
    <*fmt.wrapError | 0xc000e07c20>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-ftgghp": PUT https://rp.j4076032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ftgghp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-ftgghp\": PUT https://rp.j4076032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ftgghp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e07c00>{
            msg: "failed starting cluster creation \"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-ftgghp\": PUT https://rp.j4076032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ftgghp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Required value: field cannot be null\",\n    \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011a7080>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["206"],
                        "X-Envoy-Upstream-Service-Time": ["34"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "2542e3fe-ba0f-458c-8e87-13393388e6f3",
                        ],
                        "Date": [
                            "Tue, 24 Mar 2026 02:30:15 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000e07b20>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 206,
                    TransferEncoding: nil,
                ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
01:41:47 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d54a20>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-bhmmpv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-bhmmpv, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-bhmmpv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-bhmmpv, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001376510>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-bhmmpv",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d54a20>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-bhmmpv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-bhmmpv, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-bhmmpv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-bhmmpv, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001376510>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-bhmmpv",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 8bf5305
n/a n/a passed #4463 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 49f81d9 · merge 49f81d9
n/a n/a passed #4587 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4999eef · merge 4999eef
n/a n/a passed #4584 (merged) 0 n/a