CIHealth
<
Single Day: Oct 15▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 49m 17s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 6
e2e (6)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:16 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:198]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x371d14f72660>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-tp5x5pmzp7tp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-tp5x5pmzp7tp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 95a36805-935a-4bcb-872a-f8a4f8728f13
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:198]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x371d14f72660>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-tp5x5pmzp7tp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-tp5x5pmzp7tp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 95a36805-935a-4bcb-872a-f8a4f8728f13
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:16 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x102fe23b6c30>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9h9lz679k5f2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9h9lz679k5f2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: a95325c3-d488-4bab-80fe-86564e76d135
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x102fe23b6c30>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9h9lz679k5f2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9h9lz679k5f2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: a95325c3-d488-4bab-80fe-86564e76d135
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:16 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x5511a204930>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-jjss9n25kwrw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-jjss9n25kwrw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: 8b3df521-7631-48b4-baa7-85b52fbf1782
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x5511a204930>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-jjss9n25kwrw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-jjss9n25kwrw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: 8b3df521-7631-48b4-baa7-85b52fbf1782
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x2ca0b54fede0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-bmxkjgzxlbkg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-bmxkjgzxlbkg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: b0372a9f-398a-42d0-b5cd-edda9463ac12
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x2ca0b54fede0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-bmxkjgzxlbkg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-bmxkjgzxlbkg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: b0372a9f-398a-42d0-b5cd-edda9463ac12
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
23:00:16 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3d468a7c3d60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdw8m7lp2fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
      "name": "3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
      "status": "Failed",
      "startTime": "2026-10-15T23:18:55.172688796Z",
      "endTime": "2026-10-15T23:47:59.507731058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
            correlation ID: df77bc05-17d5-4c4c-bad3-0bd2a5dbd8cc
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3d468a7c3d60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdw8m7lp2fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
      "name": "3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
      "status": "Failed",
      "startTime": "2026-10-15T23:18:55.172688796Z",
      "endTime": "2026-10-15T23:47:59.507731058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
            correlation ID: df77bc05-17d5-4c4c-bad3-0bd2a5dbd8cc
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x1546d55ea630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-9glkpnvpmr99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-9glkpnvpmr99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: f98beb96-beea-4208-b959-4d72b7ee0a28
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x1546d55ea630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-9glkpnvpmr99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-9glkpnvpmr99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: f98beb96-beea-4208-b959-4d72b7ee0a28
            client request ID: unavailable
occurred
2h 23m 15s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth n/a passed n/a 0 n/a
2h 51m 8s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westeurope n/a passed n/a 0 n/a
2h 50m 50s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2 n/a passed n/a 0 n/a
2h 52m 47s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
12:14:00 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x7be4d399a60>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-qqvhz8j7w227" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233",
      "name": "8506345f-be03-477f-be2f-7b05e552a233",
      "status": "Failed",
      "startTime": "2026-10-15T13:54:56.565235769Z",
      "endTime": "2026-10-15T14:23:56.214949667Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
            correlation ID: 70e8c338-b9fd-439c-b234-059edc54dfd9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x7be4d399a60>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-qqvhz8j7w227" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233",
      "name": "8506345f-be03-477f-be2f-7b05e552a233",
      "status": "Failed",
      "startTime": "2026-10-15T13:54:56.565235769Z",
      "endTime": "2026-10-15T14:23:56.214949667Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
            correlation ID: 70e8c338-b9fd-439c-b234-059edc54dfd9
            client request ID: unavailable
occurred
2h 40m 44s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 centralindia n/a passed n/a 0 n/a
2h 44m 14s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 switzerlandnorth n/a passed n/a 0 n/a
2h 44m 21s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 canadacentral n/a passed n/a 0 n/a
2h 28m 44s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 2
e2e (2)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
06:27:36 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x3a1769a66d00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4m8h2dg477s7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705",
      "name": "e7a3692b-5901-4199-ae87-9018f084c705",
      "status": "Failed",
      "startTime": "2026-10-15T06:48:50.851858954Z",
      "endTime": "2026-10-15T07:17:54.261659106Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
            correlation ID: 23578d08-9577-490e-b796-4ce4da9b9b97
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x3a1769a66d00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4m8h2dg477s7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705",
      "name": "e7a3692b-5901-4199-ae87-9018f084c705",
      "status": "Failed",
      "startTime": "2026-10-15T06:48:50.851858954Z",
      "endTime": "2026-10-15T07:17:54.261659106Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
            correlation ID: 23578d08-9577-490e-b796-4ce4da9b9b97
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
06:27:36 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x25ccdbb84fe0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-fwqz99tj9vqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f",
      "name": "3b0e9b20-034d-4163-98db-b3ac4dede16f",
      "status": "Failed",
      "startTime": "2026-10-15T07:00:37.486714177Z",
      "endTime": "2026-10-15T07:29:37.711665502Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
            correlation ID: 0ad357fe-5f92-4836-9361-6bb0422c841f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x25ccdbb84fe0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-fwqz99tj9vqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f",
      "name": "3b0e9b20-034d-4163-98db-b3ac4dede16f",
      "status": "Failed",
      "startTime": "2026-10-15T07:00:37.486714177Z",
      "endTime": "2026-10-15T07:29:37.711665502Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
            correlation ID: 0ad357fe-5f92-4836-9361-6bb0422c841f
            client request ID: unavailable
occurred
2h 21m 3s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
02:58:56 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x10ea921035c0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-4426rkkk84lt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4",
      "name": "cf6cd4fd-40ae-433a-b987-569919cfb0d4",
      "status": "Failed",
      "startTime": "2026-10-15T04:26:27.342644977Z",
      "endTime": "2026-10-15T04:55:33.710462341Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
            correlation ID: 442271c1-98e4-4b63-9254-ce9229dd3320
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x10ea921035c0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-4426rkkk84lt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4",
      "name": "cf6cd4fd-40ae-433a-b987-569919cfb0d4",
      "status": "Failed",
      "startTime": "2026-10-15T04:26:27.342644977Z",
      "endTime": "2026-10-15T04:55:33.710462341Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
            correlation ID: 442271c1-98e4-4b63-9254-ce9229dd3320
            client request ID: unavailable
occurred
2h 15m 18s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 2
e2e (2)
waiting for a presented DNS-01 challenge for *
02:00:15 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.003s.
delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued
Expected success, but got an error:
    <*errors.errorString | 0x2ce43d7a8400>: 
    waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io
    {
        s: "waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.003s.
delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued
Expected success, but got an error:
    <*errors.errorString | 0x2ce43d7a8400>: 
    waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io
    {
        s: "waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io",
    }
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded
02:00:15 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
    <*fmt.wrapErrors | 0x1d3b45812a50>: 
    failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-5lx5864xtdrb" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-5lx5864xtdrb\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1d3b4591ca00>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
    <*fmt.wrapErrors | 0x1d3b45812a50>: 
    failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-5lx5864xtdrb" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-5lx5864xtdrb\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1d3b4591ca00>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred