CIHealth Failure Patterns

Environment: STG

Failure patterns: 16 · Failures matched: 92 · Total runs: 12 · Runs affected: 10

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x4fc6ce77170>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: 09054a39-11cf-4b09-a485-31e6d3a5caaf
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x4fc6ce77170>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: 09054a39-11cf-4b09-a485-31e6d3a5caaf
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2195bd58cde0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 4e9f2a97-920c-4952-a752-7a5aa59c6506
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2195bd58cde0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 4e9f2a97-920c-4952-a752-7a5aa59c6506
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x8e1dd31d290>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: bf0c2965-cddd-42e6-9d62-d3767563c18a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x8e1dd31d290>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: bf0c2965-cddd-42e6-9d62-d3767563c18a
            client request ID: unavailable
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	5

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-10-17 23:00	n/a	2111593145568333824
2026-10-16 23:00	n/a	2111230747082756096
2026-10-15 23:00	n/a	2110868388279488512
2026-10-14 23:00	n/a	2110505995703685120
2026-10-13 23:00	n/a	2110144221439594496
2026-10-12 23:00	n/a	2109781178767642624

ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed

full failure pattern:

ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x5d768dbf900>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x5d768dbf900>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x39d9d422b5e0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x39d9d422b5e0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x284d46b12080>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x284d46b12080>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	2

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-18 02:00	n/a	2111638461491449856
2026-10-17 23:00	n/a	2111593145568333824
2026-10-17 02:00	n/a	2111276046086049792
2026-10-16 23:00	n/a	2111230747082756096

VerifyHyperShiftIngressCertificate timed out after 10m0

full failure pattern:

VerifyHyperShiftIngressCertificate timed out after 10m0

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1526b2d700a0>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1526b2d70080>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1526b2d70060>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1526b2d700a0>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1526b2d70080>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1526b2d70060>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x21f4804320>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x21f48042e0>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x21f4804280>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x21f4804320>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x21f48042e0>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x21f4804280>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x27807b514a20>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x27807b5149e0>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x27807b514980>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x27807b514a20>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x27807b5149e0>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x27807b514980>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should serve the default ingress certificate for 5.1	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should serve the default ingress certificate for 5.1	2

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-18 02:00	n/a	2111638461491449856
2026-10-17 23:00	n/a	2111593145568333824
2026-10-17 02:00	n/a	2111276046086049792
2026-10-16 23:00	n/a	2111230747082756096

external auth <external-auth>.Status was nil

full failure pattern:

external auth <external-auth>.Status was nil

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.000s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
    <*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.000s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
    <*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.604s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
    <*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.604s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
    <*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.002s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
    <*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.002s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
    <*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with an external auth config and get the external auth config	2

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-18 02:00	n/a	2111638461491449856
2026-10-17 23:00	n/a	2111593145568333824
2026-10-17 02:00	n/a	2111276046086049792
2026-10-16 23:00	n/a	2111230747082756096

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-snb95l" at install version 4.22.0-0.nightly-multi-2026-10-14-193612
Unexpected error:
    <*fmt.wrapError | 0x1f800f0eba0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-snb95l: failed waiting for cluster="cp-ystream-upgrade-5-0-snb95l" in resourcegroup="rg-cp-ystream-upgrade-5-0-9pm7qw6t4kxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716",
      "name": "5b5b73d5-ad43-4248-a379-3589ba793716",
      "status": "Failed",
      "startTime": "2026-10-14T23:28:48.207254546Z",
      "endTime": "2026-10-14T23:57:58.000803929Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
            correlation ID: 8af90513-a16b-4576-aebf-be0202178446
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-snb95l" at install version 4.22.0-0.nightly-multi-2026-10-14-193612
Unexpected error:
    <*fmt.wrapError | 0x1f800f0eba0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-snb95l: failed waiting for cluster="cp-ystream-upgrade-5-0-snb95l" in resourcegroup="rg-cp-ystream-upgrade-5-0-9pm7qw6t4kxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716",
      "name": "5b5b73d5-ad43-4248-a379-3589ba793716",
      "status": "Failed",
      "startTime": "2026-10-14T23:28:48.207254546Z",
      "endTime": "2026-10-14T23:57:58.000803929Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
            correlation ID: 8af90513-a16b-4576-aebf-be0202178446
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-4bfb22rwmqzl/cluster-nightly-4-22-8zxbz4 should provision
Unexpected error:
    <*fmt.wrapError | 0x363444c03360>: 
    failed to create HCP cluster cluster-nightly-4-22-8zxbz4: failed waiting for cluster="cluster-nightly-4-22-8zxbz4" in resourcegroup="rg-nightly-4-22-4bfb22rwmqzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
      "name": "4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
      "status": "Failed",
      "startTime": "2026-10-14T23:25:15.190269818Z",
      "endTime": "2026-10-14T23:54:17.885160466Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
            correlation ID: eee8c816-0fc0-4cfa-8c09-a91c4755ee62
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-4bfb22rwmqzl/cluster-nightly-4-22-8zxbz4 should provision
Unexpected error:
    <*fmt.wrapError | 0x363444c03360>: 
    failed to create HCP cluster cluster-nightly-4-22-8zxbz4: failed waiting for cluster="cluster-nightly-4-22-8zxbz4" in resourcegroup="rg-nightly-4-22-4bfb22rwmqzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
      "name": "4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
      "status": "Failed",
      "startTime": "2026-10-14T23:25:15.190269818Z",
      "endTime": "2026-10-14T23:54:17.885160466Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
            correlation ID: eee8c816-0fc0-4cfa-8c09-a91c4755ee62
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-7c4xdncsltnp/cluster-nightly-4-21-7xdrt5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc38fe936560>: 
    failed to create HCP cluster cluster-nightly-4-21-7xdrt5: failed waiting for cluster="cluster-nightly-4-21-7xdrt5" in resourcegroup="rg-nightly-4-21-7c4xdncsltnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b",
      "name": "bc06cf13-34dc-4e88-8b6c-774417a9e32b",
      "status": "Failed",
      "startTime": "2026-10-14T23:25:05.12965498Z",
      "endTime": "2026-10-14T23:54:07.890200924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
            correlation ID: 0f756d77-a36e-4289-82ab-c5ef8b73feb4
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-7c4xdncsltnp/cluster-nightly-4-21-7xdrt5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc38fe936560>: 
    failed to create HCP cluster cluster-nightly-4-21-7xdrt5: failed waiting for cluster="cluster-nightly-4-21-7xdrt5" in resourcegroup="rg-nightly-4-21-7c4xdncsltnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b",
      "name": "bc06cf13-34dc-4e88-8b6c-774417a9e32b",
      "status": "Failed",
      "startTime": "2026-10-14T23:25:05.12965498Z",
      "endTime": "2026-10-14T23:54:07.890200924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
            correlation ID: 0f756d77-a36e-4289-82ab-c5ef8b73feb4
            client request ID: unavailable
occurred

Contributing tests (7)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-14 23:00	n/a	2110505995703685120
2026-10-13 23:00	n/a	2110144221439594496
2026-10-12 23:00	n/a	2109781178767642624

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-qfmkcj" at install version 5.0.0-0.nightly-multi-2026-10-14-152123
Unexpected error:
    <*fmt.wrapError | 0x33329457b0a0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-qfmkcj: failed waiting for cluster="cp-ystream-upgrade-5-1-qfmkcj" in resourcegroup="rg-cp-ystream-upgrade-5-1-667r4hptzmjc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0",
      "name": "b88eb811-d667-4408-bb78-6c86786b44f0",
      "status": "Failed",
      "startTime": "2026-10-14T23:29:34.061824976Z",
      "endTime": "2026-10-14T23:58:37.90403656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
            correlation ID: 8de1d04b-8264-4a06-8468-c991e9773a81
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-qfmkcj" at install version 5.0.0-0.nightly-multi-2026-10-14-152123
Unexpected error:
    <*fmt.wrapError | 0x33329457b0a0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-qfmkcj: failed waiting for cluster="cp-ystream-upgrade-5-1-qfmkcj" in resourcegroup="rg-cp-ystream-upgrade-5-1-667r4hptzmjc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0",
      "name": "b88eb811-d667-4408-bb78-6c86786b44f0",
      "status": "Failed",
      "startTime": "2026-10-14T23:29:34.061824976Z",
      "endTime": "2026-10-14T23:58:37.90403656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
            correlation ID: 8de1d04b-8264-4a06-8468-c991e9773a81
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
    <*fmt.wrapError | 0x2d73d9d10e00>: 
    failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
      "name": "41bf70ba-7344-4c62-9364-b6165697473d",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:02.793421826Z",
      "endTime": "2026-10-12T23:56:08.063137348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
            correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
    <*fmt.wrapError | 0x2d73d9d10e00>: 
    failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
      "name": "41bf70ba-7344-4c62-9364-b6165697473d",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:02.793421826Z",
      "endTime": "2026-10-12T23:56:08.063137348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
            correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
            client request ID: unavailable
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-14 23:00	n/a	2110505995703685120
2026-10-12 23:00	n/a	2109781178767642624

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x2e2df3fce3a0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-cxh9xv97rj9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be",
      "name": "0195b61d-2798-45a3-a546-b2e316a8e4be",
      "status": "Failed",
      "startTime": "2026-10-14T03:10:09.167253269Z",
      "endTime": "2026-10-14T03:39:12.28722327Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
            correlation ID: 6295da04-1f90-4641-a016-0502957f2d44
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x2e2df3fce3a0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-cxh9xv97rj9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be",
      "name": "0195b61d-2798-45a3-a546-b2e316a8e4be",
      "status": "Failed",
      "startTime": "2026-10-14T03:10:09.167253269Z",
      "endTime": "2026-10-14T03:39:12.28722327Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
            correlation ID: 6295da04-1f90-4641-a016-0502957f2d44
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x371cbc791620>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
      "name": "7da88189-d1ef-4299-b869-95f7299be069",
      "status": "Failed",
      "startTime": "2026-10-12T23:47:30.718672913Z",
      "endTime": "2026-10-13T00:16:35.706367535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
            correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x371cbc791620>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
      "name": "7da88189-d1ef-4299-b869-95f7299be069",
      "status": "Failed",
      "startTime": "2026-10-12T23:47:30.718672913Z",
      "endTime": "2026-10-13T00:16:35.706367535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
            correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
            client request ID: unavailable
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-14 02:00	n/a	2110189439996137472
2026-10-12 23:00	n/a	2109781178767642624

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
    <*fmt.wrapError | 0x15168f06cae0>: 
    failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
      "name": "8479abf5-39da-412b-bb04-87374bf3b757",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:19.685007768Z",
      "endTime": "2026-10-13T01:14:27.351105107Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
            correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
    <*fmt.wrapError | 0x15168f06cae0>: 
    failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
      "name": "8479abf5-39da-412b-bb04-87374bf3b757",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:19.685007768Z",
      "endTime": "2026-10-13T01:14:27.351105107Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
            correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3152b3c1a500>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:30.036006045Z",
      "endTime": "2026-10-13T01:14:37.44566298Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
            correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3152b3c1a500>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:30.036006045Z",
      "endTime": "2026-10-13T01:14:37.44566298Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
            correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x15a1e46902c0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
      "name": "8f006a7a-53bf-4566-93d0-8202c11da346",
      "status": "Failed",
      "startTime": "2026-10-13T00:52:36.6753625Z",
      "endTime": "2026-10-13T01:21:37.350837076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
            correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x15a1e46902c0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
      "name": "8f006a7a-53bf-4566-93d0-8202c11da346",
      "status": "Failed",
      "startTime": "2026-10-13T00:52:36.6753625Z",
      "endTime": "2026-10-13T01:21:37.350837076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
            correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
            client request ID: unavailable
occurred

Contributing tests (11)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-13 00:11	n/a	2109799288455303168

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenSh...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1854e4898c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1854e4898c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x13a151ae2450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
            correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x13a151ae2450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
            correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x1d06db38a6c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x1d06db38a6c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
            client request ID: unavailable
occurred

Contributing tests (10)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-13 00:11	n/a	2109799288455303168

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenS...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x31faef804060>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
            correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x31faef804060>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
            correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x258acfd8b940>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
            correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x258acfd8b940>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
            correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1b553dfaa40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
            correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1b553dfaa40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
            correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
            client request ID: unavailable
occurred

Contributing tests (9)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-13 00:11	n/a	2109799288455303168

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpe...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x2299302636e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
            correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x2299302636e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
            correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x33ccd41339b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
            correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x33ccd41339b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
            correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x1206c2abaf90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
            correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x1206c2abaf90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
            correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
            client request ID: unavailable
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-13 00:11	n/a	2109799288455303168

CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
    <*fmt.wrapErrors | 0x95f8488fc80>: 
    failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-n6c4c4fd2ssk" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-n6c4c4fd2ssk\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x95f84458c30>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
    <*fmt.wrapErrors | 0x95f8488fc80>: 
    failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-n6c4c4fd2ssk" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-n6c4c4fd2ssk\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x95f84458c30>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-14 02:00	n/a	2110189439996137472

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
    <*fmt.wrapError | 0xc4ca110a7e0>: 
    failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:12.849511271Z",
      "endTime": "2026-10-12T23:56:15.713757928Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
            correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
    <*fmt.wrapError | 0xc4ca110a7e0>: 
    failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:12.849511271Z",
      "endTime": "2026-10-12T23:56:15.713757928Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
            correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-12 23:00	n/a	2109781178767642624

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-zxqcgtfv7nf5/cluster-nightly-5-1-2dz545 should provision
Unexpected error:
    <*fmt.wrapError | 0x3ec63f648620>: 
    failed to create HCP cluster cluster-nightly-5-1-2dz545: failed waiting for cluster="cluster-nightly-5-1-2dz545" in resourcegroup="rg-nightly-5-1-zxqcgtfv7nf5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
      "name": "c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
      "status": "Failed",
      "startTime": "2026-10-14T23:26:17.571795133Z",
      "endTime": "2026-10-14T23:55:18.071165518Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, catalog-operator, packageserver, openshift-controller-manager, dns-operator, openshift-route-controller-manager, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
            correlation ID: b45fa2ae-f53a-4948-b0fb-190aed3be0b8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-zxqcgtfv7nf5/cluster-nightly-5-1-2dz545 should provision
Unexpected error:
    <*fmt.wrapError | 0x3ec63f648620>: 
    failed to create HCP cluster cluster-nightly-5-1-2dz545: failed waiting for cluster="cluster-nightly-5-1-2dz545" in resourcegroup="rg-nightly-5-1-zxqcgtfv7nf5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
      "name": "c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
      "status": "Failed",
      "startTime": "2026-10-14T23:26:17.571795133Z",
      "endTime": "2026-10-14T23:55:18.071165518Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, catalog-operator, packageserver, openshift-controller-manager, dns-operator, openshift-route-controller-manager, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
            correlation ID: b45fa2ae-f53a-4948-b0fb-190aed3be0b8
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-14 23:00	n/a	2110505995703685120

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-8qt7kcrvhtlb/cluster-nightly-5-0-6wsq4v should provision
Unexpected error:
    <*fmt.wrapError | 0x30f3685a760>: 
    failed to create HCP cluster cluster-nightly-5-0-6wsq4v: failed waiting for cluster="cluster-nightly-5-0-6wsq4v" in resourcegroup="rg-nightly-5-0-8qt7kcrvhtlb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270",
      "name": "70ee1e74-bffb-4dec-89fb-282756ca2270",
      "status": "Failed",
      "startTime": "2026-10-14T23:26:07.782117382Z",
      "endTime": "2026-10-14T23:55:07.903819031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-0.nightly-multi-2026-10-14-152123 is Partial (want Completed), started 20m19s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
            correlation ID: e153794f-fb7c-4e3b-9441-a5dadf7b0225
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-8qt7kcrvhtlb/cluster-nightly-5-0-6wsq4v should provision
Unexpected error:
    <*fmt.wrapError | 0x30f3685a760>: 
    failed to create HCP cluster cluster-nightly-5-0-6wsq4v: failed waiting for cluster="cluster-nightly-5-0-6wsq4v" in resourcegroup="rg-nightly-5-0-8qt7kcrvhtlb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270",
      "name": "70ee1e74-bffb-4dec-89fb-282756ca2270",
      "status": "Failed",
      "startTime": "2026-10-14T23:26:07.782117382Z",
      "endTime": "2026-10-14T23:55:07.903819031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-0.nightly-multi-2026-10-14-152123 is Partial (want Completed), started 20m19s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
            correlation ID: e153794f-fb7c-4e3b-9441-a5dadf7b0225
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-14 23:00	n/a	2110505995703685120

VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized

full failure pattern:

VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x3d800eb17200>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3d800f297260>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3d800f297240>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e191e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3d800e204000>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3d800eb05ee0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e0fab40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x3d800eb17200>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3d800f297260>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3d800f297240>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e191e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3d800e204000>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3d800eb05ee0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e0fab40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should not be able to perform various invalid operations on cluster resources	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-12 23:00	n/a	2109781178767642624