CIHealth
<
Single Day: Oct 11▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 45m 47s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 6
e2e (6)
VerifyBasicAccess failed: failed to list services: Unauthorized
23:00:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
    <*errors.joinError | 0x19c6c82ad0e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x19c6c78ffea0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c78ffe80>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7ea8960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x19c6c8695740>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c8695720>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7b65c20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
    <*errors.joinError | 0x19c6c82ad0e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x19c6c78ffea0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c78ffe80>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7ea8960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x19c6c8695740>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c8695720>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7b65c20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:04 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2478de214db0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-9589trzrcm22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-9589trzrcm22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 1985dc52-0e75-413c-9ede-87cb3e6ad675
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2478de214db0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-9589trzrcm22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-9589trzrcm22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 1985dc52-0e75-413c-9ede-87cb3e6ad675
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:04 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2482f98dd8f0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-kgfhn26xlnc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-kgfhn26xlnc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: 6cddad86-b2cb-402b-b9c5-2d33d805daed
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2482f98dd8f0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-kgfhn26xlnc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-kgfhn26xlnc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: 6cddad86-b2cb-402b-b9c5-2d33d805daed
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:04 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x1fc144a8a360>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-48s6s8v68r22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-48s6s8v68r22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 308b2783-d950-49dc-bba1-b7f5095895de
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x1fc144a8a360>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-48s6s8v68r22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-48s6s8v68r22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 308b2783-d950-49dc-bba1-b7f5095895de
            client request ID: unavailable
occurred
VerifyBasicAccess failed: failed to list services: Unauthorized
23:00:04 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x11944cc8930>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x11944de5c00>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x11944de5be0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x11945431e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x11944978520>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x11944978500>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x11944c97ae0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x11944cc8930>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x11944de5c00>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x11944de5be0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x11945431e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x11944978520>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x11944978500>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x11944c97ae0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x1809749e4ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-bt79q5smg9ln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-bt79q5smg9ln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 169674f2-a45c-486a-9ed8-ef274e1803f2
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x1809749e4ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-bt79q5smg9ln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-bt79q5smg9ln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 169674f2-a45c-486a-9ed8-ef274e1803f2
            client request ID: unavailable
occurred
2h 37m 10s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
22:05:45 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x5325624e738>: 
    resource group "cni-cilium-g76pf6t9h842": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-g76pf6t9h842" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84",
      "name": "179cc551-21d3-4583-b554-f672e37b3a84",
      "status": "Failed",
      "startTime": "2026-10-11T22:50:53.504973281Z",
      "endTime": "2026-10-11T23:14:57.954117538Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tcruuiers9mtsh6kof4evdrtdk9p3e4 still exists (deletion dispatched at 2026-10-11T22:50:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x5325624e738>: 
    resource group "cni-cilium-g76pf6t9h842": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-g76pf6t9h842" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84",
      "name": "179cc551-21d3-4583-b554-f672e37b3a84",
      "status": "Failed",
      "startTime": "2026-10-11T22:50:53.504973281Z",
      "endTime": "2026-10-11T23:14:57.954117538Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tcruuiers9mtsh6kof4evdrtdk9p3e4 still exists (deletion dispatched at 2026-10-11T22:50:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 19m 34s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a