CIHealth Failure Patterns

Environment: DEV

Failure patterns: 119 · Failures matched: 1726 · Total runs: 407 · Runs affected: 296

Failure pattern

Failed at

Trend

Also in

ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsof...

full failure pattern:

ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x14550d98c9a0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-p98fbdr6lfb6": PUT https://rp.j6887424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p98fbdr6lfb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x14550d98c9a0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-p98fbdr6lfb6": PUT https://rp.j6887424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p98fbdr6lfb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-10-09 11:01	PR #7397	2108513285430054912
2026-10-09 07:02	PR #7397	2108452968792592384
2026-10-08 22:00	PR #7397	2108316538086887424
2026-10-08 12:51	PR #7397	2108178486299463680
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 03:55	PR #7397	2107681070840287232

VerifyHyperShiftIngressCertificate timed out after 10m0

full failure pattern:

VerifyHyperShiftIngressCertificate timed out after 10m0

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.1	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-08 07:02	PR #7325	2108090608831696896
2026-10-08 02:14	PR #7325	2108018145892503552
2026-10-07 23:05	PR #7325	2107970662109810688

failed to create nodepool <nodepool>: failed starting nodepool creation "np-1" for cluster "<cluster>" in resourcegroup="<resource-group>": PUT <url>

full failure pattern:

failed to create nodepool <nodepool>: failed starting nodepool creation "np-1" for cluster "<cluster>" in resourcegroup="<resource-group>": PUT <url>

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e182923c240>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ls445v45mtp4": PUT https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-ls445v45mtp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e182923c240>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ls445v45mtp4": PUT https://rp.j9599616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-ls445v45mtp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:162]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e9eea24a6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fh65w65xjrlv": PUT https://rp.j8091008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fh65w65xjrlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:162]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2e9eea24a6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fh65w65xjrlv": PUT https://rp.j8091008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fh65w65xjrlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21.36\": node pool version 4.21.36 cannot coexist with control plane version 5.1.0-ec.1",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.1	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 12:57	PR #7325	2107817608098091008

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced; ...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x1d7c60a347e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-wh6shv999rvk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "name": "d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.15141187Z",
      "endTime": "2026-10-09T18:31:17.847497361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x1d7c60a347e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-wh6shv999rvk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "name": "d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.15141187Z",
      "endTime": "2026-10-09T18:31:17.847497361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3aa3a58fd700>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9mpb5rglfkx" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "name": "501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.983214688Z",
      "endTime": "2026-10-09T18:31:17.777246897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3aa3a58fd700>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9mpb5rglfkx" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "name": "501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.983214688Z",
      "endTime": "2026-10-09T18:31:17.777246897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x3f8cfca18160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-d9vjk88cdxmv" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "name": "715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:40.105550226Z",
      "endTime": "2026-10-09T18:31:47.946839601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x3f8cfca18160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-d9vjk88cdxmv" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "name": "715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:40.105550226Z",
      "endTime": "2026-10-09T18:31:47.946839601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (66)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-09 17:15	PR #7440	2108607415241936896

ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])

full failure pattern:

ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:142]: Timed out after 1200.007s.
Insights Operator did not reach Disabled/NoToken state, the NoToken-to-healthy transition cannot be tested
Expected success, but got an error:
    <*errors.errorString | 0x27811265a7d0>: 
    ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
    {
        s: "ClusterOperator \"insights\" not yet in Disabled/NoToken state (conditions: [])",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:142]: Timed out after 1200.007s.
Insights Operator did not reach Disabled/NoToken state, the NoToken-to-healthy transition cannot be tested
Expected success, but got an error:
    <*errors.errorString | 0x27811265a7d0>: 
    ClusterOperator "insights" not yet in Disabled/NoToken state (conditions: [])
    {
        s: "ClusterOperator \"insights\" not yet in Disabled/NoToken state (conditions: [])",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to see an HCP cluster registered with console.redhat.com via the Insights Operator	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 21:02	PR #6954	2107939651057618944

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-<id> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x2952d6db0d20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-ksddfkp5xbq5" to finish creating: GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "name": "6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "status": "Failed",
      "startTime": "2026-10-11T16:48:04.203625548Z",
      "endTime": "2026-10-11T17:17:06.363062447Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkw9f-sxlv4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkw9f-sxlv4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
            correlation ID: 9211dc6b-1604-468e-b90e-8381473929df
            client request ID: 967f10bc-a33e-4e81-877c-ed8ff3004e75
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x2952d6db0d20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-ksddfkp5xbq5" to finish creating: GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "name": "6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "status": "Failed",
      "startTime": "2026-10-11T16:48:04.203625548Z",
      "endTime": "2026-10-11T17:17:06.363062447Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkw9f-sxlv4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkw9f-sxlv4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
            correlation ID: 9211dc6b-1604-468e-b90e-8381473929df
            client request ID: 967f10bc-a33e-4e81-877c-ed8ff3004e75
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-11 15:50	PR #6746	2109310669768626176

ERROR CODE: DeploymentFailed; detail code BadRequest; detail message The requested operation cannot be performed because the database account arohcpci01-rp-j1672192 is in the proc...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code BadRequest; detail message The requested operation cannot be performed because the database account arohcpci01-rp-j1672192 is in the process of being created. Please wait until account creation completes and retry the operation.\\\\\\\ \\\\\\ ActivityId: <uuid>, Microsoft.Azure.Documents.Common/2.14.0"\\\ \\ }; provider Microsoft.DocumentDB

Signal:

provision

none

Full failure examples (1)

time=2026-10-05T23:36:27.137Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-05T23:36:28.752Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-05T23:36:29.424Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=5dd69a04f62785b0df4182f2e62bb16930ed2731284e7680f8dc157ea0096d63 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1672192%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5dd69a04f62785b0df4182f2e62bb16930ed2731284e7680f8dc157ea0096d63
time=2026-10-05T23:38:29.834Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1672192/providers/Microsoft.Resources/deployments/5dd69a04f62785b0df4182f2e62bb16930ed2731284e7680f8dc157ea0096d63/operationStatuses/08584103634964314871\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1672192/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1672192/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j1672192\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"BadRequest\\\\\\\",\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"The requested operation cannot be performed because the database account arohcpci01-rp-j1672192 is in the process of being created. Please wait until account creation completes and retry the operation.\\\\\\\\r\\\\\\\\nActivityId: 75b0de9b-7548-4bae-95bc-f97589468144, Microsoft.Azure.Documents.Common/2.14.0\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 23:04	PR #7105	2107245692081672192

ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Resources/deployments/profile-thqgrr3ymebbw' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '<timestamp>' with correlationId '<uuid>', and will expire at '<timestamp>' if it does not complete before then.

Signal:

provision

none

Full failure examples (1)

time=2026-10-05T17:12:22.862Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-05T17:12:23.287Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-10-05T17:12:24.013Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=e2e1fa40714c294cc444d45df7d65e7230bd956473842bf5c4f49c5bfba93caa portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9265280-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fe2e1fa40714c294cc444d45df7d65e7230bd956473842bf5c4f49c5bfba93caa
time=2026-10-05T17:13:24.301Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9265280-mgmt-2/providers/Microsoft.Resources/deployments/e2e1fa40714c294cc444d45df7d65e7230bd956473842bf5c4f49c5bfba93caa/operationStatuses/08584103865419476507\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"DeploymentActive\\\",\\r\\n    \\\"message\\\": \\\"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9265280-mgmt-2/providers/Microsoft.Resources/deployments/profile-thqgrr3ymebbw' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/5/2026 5:12:37 PM' with correlationId '9bd28390-2035-44ef-a1dc-d068a232c592', and will expire at '10/12/2026 5:12:37 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 16:27	PR #7360	2107145790039265280

ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/user...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-puller' was not found.; provider Microsoft.ManagedIdentity

Signal:

provision

none

Full failure examples (1)

time=2026-10-05T09:59:52.082Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-10-05T09:59:52.761Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-10-05T09:59:53.566Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=c9397b972a000626b7c9dd06c82246169ec346b2e88665f7f2672f34e557c5c2 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2945792-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc9397b972a000626b7c9dd06c82246169ec346b2e88665f7f2672f34e557c5c2
time=2026-10-05T10:01:54.159Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2945792-mgmt-2/providers/Microsoft.Resources/deployments/c9397b972a000626b7c9dd06c82246169ec346b2e88665f7f2672f34e557c5c2/operationStatuses/08584104124922522992\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2945792-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-post\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"NotFound\\\",\\r\\n            \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2945792-mgmt-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-puller' was not found.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 09:25	PR #6746	2107039414742945792

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3974085580d8>: 
    resource group "external-auth-cluster-jkpp8mdvzxsh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-jkpp8mdvzxsh" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac",
      "name": "7ef42875-c74c-425c-8fe8-0909778dceac",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:50.350274496Z",
      "endTime": "2026-10-07T18:24:57.82988818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3adt66dtiffarpvt6smilc2f19037 still exists (deletion dispatched at 2026-10-07T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3974085580d8>: 
    resource group "external-auth-cluster-jkpp8mdvzxsh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-jkpp8mdvzxsh" to finish deleting: GET https://rp.j3305728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ef42875-c74c-425c-8fe8-0909778dceac",
      "name": "7ef42875-c74c-425c-8fe8-0909778dceac",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:50.350274496Z",
      "endTime": "2026-10-07T18:24:57.82988818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3adt66dtiffarpvt6smilc2f19037 still exists (deletion dispatched at 2026-10-07T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:49	PR #7143	2107875990943305728

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f397df201c8>: 
    resource group "clusternp128-89h59bcbsmnq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-89h59bcbsmnq" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "name": "84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:09.863415259Z",
      "endTime": "2026-10-07T18:24:11.307794702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3chch61ipf94injnrljj1hk0caf92 still exists (deletion dispatched at 2026-10-07T18:00:10Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f397df201c8>: 
    resource group "clusternp128-89h59bcbsmnq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-89h59bcbsmnq" to finish deleting: GET https://rp.j6658560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "name": "84ac2f7c-19a8-4b2d-82b9-6595d5a024a0",
      "status": "Failed",
      "startTime": "2026-10-07T18:00:09.863415259Z",
      "endTime": "2026-10-07T18:24:11.307794702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ta3chch61ipf94injnrljj1hk0caf92 still exists (deletion dispatched at 2026-10-07T18:00:10Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 16:47	PR #7400	2107875420446658560

ERROR CODE: InvalidRequestContent; detail message Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/<subscription>/resourcegroups/<resource-gro...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Monitor/accounts/hcps-j2198528,/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Monitor/accounts/services-j2198528; got: /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Monitor/accounts/services-j2198528; provider Microsoft.Monitor

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0x383c4f08c1e0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/hcps-j2198528,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528; got: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528"
        }
    }
    {
        s: "expected status 200, got 400: {\n    \"error\": {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/hcps-j2198528,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528; got: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528\"\n    }\n}",
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0x383c4f08c1e0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/hcps-j2198528,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528; got: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528"
        }
    }
    {
        s: "expected status 200, got 400: {\n    \"error\": {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/hcps-j2198528,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528; got: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2198528/providers/Microsoft.Monitor/accounts/services-j2198528\"\n    }\n}",
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to perform alert processing rule CRUD operations	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 16:07	PR #7310	2107140685202198528

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The resource 'Microsoft.ManagedIdentity/userAssignedIdentities/<resource>' at line '<line>' and...

full failure pattern:

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The resource 'Microsoft.ManagedIdentity/userAssignedIdentities/<resource>' at line '<line>' and column '<column>' is defined multiple times in a template.; provider Microsoft.ManagedIdentity

Signal:

provision

none

Full failure examples (1)

time=2026-10-06T08:32:52.467Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-06T08:32:54.158Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-06T08:32:54.974Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0123648-svc/providers/Microsoft.Resources/deployments/dc9f1e5996856bf7741adb87173ebf33087f59e0051caf5ece6b3f88211e9997\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplate\",\n    \"message\": \"Deployment template validation failed: 'The resource 'Microsoft.ManagedIdentity/userAssignedIdentities/ci01-j0123648-svc-msi' at line '1' and column '101637' is defined multiple times in a template. Please see https://aka.ms/arm-syntax-resources for usage details.'.\",\n    \"additionalInfo\": [\n      {\n        \"type\": \"TemplateViolation\",\n        \"info\": {\n          \"lineNumber\": 1,\n          \"linePosition\": 101637,\n          \"path\": \"\"\n        }\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 08:07	PR #7322	2107382338580123648

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The template variable 'identityResourceGroup' is not valid: The language expression property ar...

full failure pattern:

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The template variable 'identityResourceGroup' is not valid: The language expression property array index '0' is out of bounds..

Signal:

provision

none

Full failure examples (1)

time=2026-10-06T10:39:43.882Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mgmt-agent-output stamp=1 description="Step mgmt-agent-output\n  Kind: ARM\n  Template: modules/mgmt-agent/mgmt-agent-lookup.bicep\n  Parameters: configurations/mgmt-agent-lookup.tmpl.bicepparam"
time=2026-10-06T10:39:44.398Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mgmt-agent-output stamp=1
time=2026-10-06T10:39:44.832Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mgmt-agent-output stamp=1 err="stamp 1: failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3359104-mgmt-1/providers/Microsoft.Resources/deployments/d101131e0ac63c3b52857e992498ed87773d66d0b73ed83fc948f011d3d36a38\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplate\",\n    \"message\": \"Deployment template validation failed: 'The template variable 'identityResourceGroup' is not valid: The language expression property array index '0' is out of bounds.. Please see https://aka.ms/arm-functions for usage details.'.\",\n    \"additionalInfo\": [\n      {\n        \"type\": \"TemplateViolation\",\n        \"info\": {\n          \"lineNumber\": 1,\n          \"linePosition\": 1404,\n          \"path\": \"properties.template.functions[0].members.getManagementIdentityResourceGroup.output.value\"\n        }\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mgmt-agent-output (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 10:06	PR #7322	2107412275693359104

Internal error occurred: failed calling webhook "webhook

full failure pattern:

Internal error occurred: failed calling webhook "webhook

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:337]: Timed out after 120.000s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
    <*errors.StatusError | 0x3dfc585d0640>: 
    Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:337]: Timed out after 120.000s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
    <*errors.StatusError | 0x3dfc585d0640>: 
    Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 08:29	PR #7370	2107750168211230720

Unauthorized

full failure pattern:

Unauthorized

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:128]: failed to disable kube-proxy via network operator patch
Unexpected error:
    <*errors.StatusError | 0x1122950dbe00>: 
    Unauthorized
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Unauthorized",
            Reason: "Unauthorized",
            Details: nil,
            Code: 401,
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:128]: failed to disable kube-proxy via network operator patch
Unexpected error:
    <*errors.StatusError | 0x1122950dbe00>: 
    Unauthorized
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
                ShardInfo: nil,
            },
            Status: "Failure",
            Message: "Unauthorized",
            Reason: "Unauthorized",
            Details: nil,
            Code: 401,
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 22:59	PR #7331	2107244242656038912

VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout

full failure pattern:

VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:128]: failed to verify HCP cluster viability
Unexpected error:
    <*errors.joinError | 0x3fa9a7564d98>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0x3fa9a6cff3e0>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6cff3c0>{
                    msg: "failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a7398db0>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0x3fa9a716a690>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3fa9a6ee5c60>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6ee5c40>{
                    msg: "failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a786a420>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0x3fa9a7c7d0e0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:128]: failed to verify HCP cluster viability
Unexpected error:
    <*errors.joinError | 0x3fa9a7564d98>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0x3fa9a6cff3e0>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6cff3c0>{
                    msg: "failed to list services: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a7398db0>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0x3fa9a716a690>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3fa9a6ee5c60>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0x3fa9a6ee5c40>{
                    msg: "failed to list all APIServices: Get \"https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0x3fa9a786a420>{
                        Op: "Get",
                        URL: "https://api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0x3fa9a7c7d0e0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0x3fa9a7c7cfa0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.157.3:34489->172.30.0.10:53: i/o timeout",
                                Name: "api.g0o4h9l1p1i8t8c.5oub.j2737280.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 03:35	PR #7400	2107676117312737280

VerifyDefaultIngressCertificate timed out after 35m0.001s: default ingress route must serve trusted HTTPS: Get "<url>": context deadline exceeded

full failure pattern:

VerifyDefaultIngressCertificate timed out after 35m0.001s: default ingress route must serve trusted HTTPS: Get "<url>": context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_serve_default_ingress_certificate_public.go:178]: sample app must return HTTP 200 over trusted HTTPS through public ingress
Unexpected error:
    <*fmt.wrapError | 0x2e2614a7c240>: 
    VerifyDefaultIngressCertificate timed out after 35m0.001s: default ingress route must serve trusted HTTPS: Get "https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud": context deadline exceeded
    {
        msg: "VerifyDefaultIngressCertificate timed out after 35m0.001s: default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud\": context deadline exceeded",
        err: <*fmt.wrapError | 0x2e2614a7ce60>{
            msg: "default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud\": context deadline exceeded",
            err: <*url.Error | 0x2e261602fad0>{
                Op: "Get",
                URL: "https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud",
                Err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_serve_default_ingress_certificate_public.go:178]: sample app must return HTTP 200 over trusted HTTPS through public ingress
Unexpected error:
    <*fmt.wrapError | 0x2e2614a7c240>: 
    VerifyDefaultIngressCertificate timed out after 35m0.001s: default ingress route must serve trusted HTTPS: Get "https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud": context deadline exceeded
    {
        msg: "VerifyDefaultIngressCertificate timed out after 35m0.001s: default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud\": context deadline exceeded",
        err: <*fmt.wrapError | 0x2e2614a7ce60>{
            msg: "default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud\": context deadline exceeded",
            err: <*url.Error | 0x2e261602fad0>{
                Op: "Get",
                URL: "https://agnhost-e2e-sample-app-r8lfk.apps.aro.r7l3x8z0s2v8m3o.7lbv.j5366528.hcp.osadev.cloud",
                Err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate through public ingress with OCP candidate channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 03:56	PR #7325	2107319014425366528

VerifyDefaultIngressCertificate timed out after 35m0.003s: default ingress route must serve trusted HTTPS: Get "<url>": context deadline exceeded

full failure pattern:

VerifyDefaultIngressCertificate timed out after 35m0.003s: default ingress route must serve trusted HTTPS: Get "<url>": context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_serve_default_ingress_certificate_public.go:162]: sample app must return HTTP 200 over trusted HTTPS through public ingress
Unexpected error:
    <*fmt.wrapError | 0x1ec2e1beb2e0>: 
    VerifyDefaultIngressCertificate timed out after 35m0.003s: default ingress route must serve trusted HTTPS: Get "https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud": context deadline exceeded
    {
        msg: "VerifyDefaultIngressCertificate timed out after 35m0.003s: default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1ec2e1beafe0>{
            msg: "default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud\": context deadline exceeded",
            err: <*url.Error | 0x1ec2e1a4fb60>{
                Op: "Get",
                URL: "https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud",
                Err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_serve_default_ingress_certificate_public.go:162]: sample app must return HTTP 200 over trusted HTTPS through public ingress
Unexpected error:
    <*fmt.wrapError | 0x1ec2e1beb2e0>: 
    VerifyDefaultIngressCertificate timed out after 35m0.003s: default ingress route must serve trusted HTTPS: Get "https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud": context deadline exceeded
    {
        msg: "VerifyDefaultIngressCertificate timed out after 35m0.003s: default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1ec2e1beafe0>{
            msg: "default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud\": context deadline exceeded",
            err: <*url.Error | 0x1ec2e1a4fb60>{
                Op: "Get",
                URL: "https://agnhost-e2e-sample-app-twj87.apps.aro.r5v0b5h5l4z1l6s.jobe.j9797888.hcp.osadev.cloud",
                Err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate through public ingress with OCP candidate channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 20:28	PR #7325	2107206247269797888

VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-<id> on node <node> after retries: a...

full failure pattern:

VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-<id> on node <node> after retries: an error on the server ("unknown") has prevented the request from succeeding

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xf6f46a980d8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-tkqcj)
    {
        errs: [
            <*fmt.wrapError | 0xf6f45174120>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                err: <*fmt.wrapError | 0xf6f45174100>{
                    msg: "failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                    err: <*fmt.wrapError | 0xf6f451742a0>{
                        msg: "failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                        err: <*errors.StatusError | 0xf6f4458f7c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-tkqcj",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xf6f46a980d8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-tkqcj)
    {
        errs: [
            <*fmt.wrapError | 0xf6f45174120>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                err: <*fmt.wrapError | 0xf6f45174100>{
                    msg: "failed to check FIPS mode on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p: failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                    err: <*fmt.wrapError | 0xf6f451742a0>{
                        msg: "failed to get pod logs for pod fips-check-tkqcj in namespace e2e-fips-test-zn5bh on node h1f2m2p1y3c2t8a-np-1-5mzcd-8fr8p after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                        err: <*errors.StatusError | 0xf6f4458f7c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-tkqcj)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-tkqcj",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 19:32	PR #7325	2107916955439599616

VerifyImagePulled(acrpull6864z6s54xlk.azurecr.io/ubi-minimal) timed out after 2m0.006s: expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-d...

full failure pattern:

VerifyImagePulled(acrpull6864z6s54xlk.azurecr.io/ubi-minimal) timed out after 2m0.006s: expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpull6864z6s54xlk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x88c3b4bd360>: 
    VerifyImagePulled(acrpull6864z6s54xlk.azurecr.io/ubi-minimal) timed out after 2m0.006s: expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull6864z6s54xlk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3e6da004-1a48-4c61-b7cd-2c06205130ad; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c421171-6e86-4b5b-8da9-c36681c3d54a
    {
        msg: "VerifyImagePulled(acrpull6864z6s54xlk.azurecr.io/ubi-minimal) timed out after 2m0.006s: expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n  - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull6864z6s54xlk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3e6da004-1a48-4c61-b7cd-2c06205130ad; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c421171-6e86-4b5b-8da9-c36681c3d54a",
        err: <*errors.errorString | 0x88c3b30f800>{
            s: "expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n  - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull6864z6s54xlk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3e6da004-1a48-4c61-b7cd-2c06205130ad; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c421171-6e86-4b5b-8da9-c36681c3d54a",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpull6864z6s54xlk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x88c3b4bd360>: 
    VerifyImagePulled(acrpull6864z6s54xlk.azurecr.io/ubi-minimal) timed out after 2m0.006s: expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull6864z6s54xlk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3e6da004-1a48-4c61-b7cd-2c06205130ad; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c421171-6e86-4b5b-8da9-c36681c3d54a
    {
        msg: "VerifyImagePulled(acrpull6864z6s54xlk.azurecr.io/ubi-minimal) timed out after 2m0.006s: expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n  - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull6864z6s54xlk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3e6da004-1a48-4c61-b7cd-2c06205130ad; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c421171-6e86-4b5b-8da9-c36681c3d54a",
        err: <*errors.errorString | 0x88c3b30f800>{
            s: "expected image from acrpull6864z6s54xlk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n  - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull6864z6s54xlk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3e6da004-1a48-4c61-b7cd-2c06205130ad; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c421171-6e86-4b5b-8da9-c36681c3d54a",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 23:05	PR #7313	2107245793541885952

alert [hcp] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [hcp] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-05T12:54:04Z
  Ended:       2026-10-05T13:00:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2217856-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.214:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2217856/providers/microsoft.monitor/accounts/hcps-j2217856", namespace="ocm-arohcpci01-2t8kbgro552tj4iv3ti1u3mhvuvqkpb8-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-twm4t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2t8kbgro552tj4iv3ti1u3mhvuvqkpb8-ksm-hcp-metrics/capi-provider is not progressing for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-05T12:54:04Z
  Ended:       2026-10-05T13:00:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j2217856-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.214:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2217856/providers/microsoft.monitor/accounts/hcps-j2217856", namespace="ocm-arohcpci01-2t8kbgro552tj4iv3ti1u3mhvuvqkpb8-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-twm4t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2t8kbgro552tj4iv3ti1u3mhvuvqkpb8-ksm-hcp-metrics/capi-provider is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 11:11	PR #7349	2107066190932217856

context deadline exceeded. To troubleshoot, visit <url>

full failure pattern:

context deadline exceeded. To troubleshoot, visit <url>

Signal:

e2e

none

Full failure examples (1)

github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0x17a8dd744af0?, {0x7f91c02f24c0?, 0x17a8dd261560?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1075 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0x17a8dcfdf200, {0x7f91c02f24c0, 0x17a8dd261560}, {0xf9f89cb, 0x15}, {0x17a8dc60e129, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:508 +0x2a8
github.com/Azure/ARO-HCP/test/e2e.init.func34.2({0x7f91c02f24c0, 0x17a8dd261560})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:68 +0x139
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x101a3e60?, 0x17a8dd261560?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 134
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xdfb

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked: failed to get subscription ID: ClientSecretCredential: unable to resolve an endpoint: server response error:
 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0x17a8dd744af0?, {0x7f91c02f24c0?, 0x17a8dd261560?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1075 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0x17a8dcfdf200, {0x7f91c02f24c0, 0x17a8dd261560}, {0xf9f89cb, 0x15}, {0x17a8dc60e129, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:508 +0x2a8
github.com/Azure/ARO-HCP/test/e2e.init.func34.2({0x7f91c02f24c0, 0x17a8dd261560})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:68 +0x139
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x101a3e60?, 0x17a8dd261560?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 134
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xdfb

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked: failed to get subscription ID: ClientSecretCredential: unable to resolve an endpoint: server response error:
 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 16:13	PR #7232	2108229382379147264

error for disabled boot diagnostics should contain 409 status code

full failure pattern:

error for disabled boot diagnostics should contain 409 status code

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:456]: error for disabled boot diagnostics should contain 409 status code
Expected
    <string>: failed to send request: Get "https://admin.j1851392.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-mx7j6swwr25v/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster-bootdiag/serialconsole?vmName=d3i6t7c9q4h0k8j-worker-wrlpp-npg85": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
to contain substring
    <string>: 409
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:456]: error for disabled boot diagnostics should contain 409 status code
Expected
    <string>: failed to send request: Get "https://admin.j1851392.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-mx7j6swwr25v/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster-bootdiag/serialconsole?vmName=d3i6t7c9q4h0k8j-worker-wrlpp-npg85": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
to contain substring
    <string>: 409

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-09 01:49	PR #7437	2108374328641851392

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j2345472-mgmt-2","clusterName":"ci01-j234547

Signal:

provision

none

Full failure examples (1)

time=2026-10-06T07:49:49.352Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-06T08:19:48.987Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-06T07:49:49.36213735Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2345472-mgmt-2\",\"clusterName\":\"ci01-j2345472-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-10-06T07:49:51.126744886Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2345472-mgmt-2\",\"clusterName\":\"ci01-j2345472-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-06T07:49:51.49238422Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2345472-mgmt-2\",\"clusterName\":\"ci01-j2345472-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 07:20	PR #7332	2107370522202345472

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j4452224-mgmt-1","clusterName":"ci01-j445222

Signal:

provision

none

Full failure examples (1)

time=2026-10-08T14:32:31.059Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T15:02:30.832Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T14:32:31.068088679Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T14:32:32.982339672Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T14:32:33.61885433Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 13:42	PR #7427	2108191395574452224

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j4987648-mgmt-1","clusterName":"ci01-j498764

Signal:

provision

none

Full failure examples (1)

time=2026-10-06T11:09:01.284Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-06T11:39:00.940Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-06T11:09:01.295609297Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4987648-mgmt-1\",\"clusterName\":\"ci01-j4987648-mgmt-1\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-10-06T11:09:03.113200077Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4987648-mgmt-1\",\"clusterName\":\"ci01-j4987648-mgmt-1\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-06T11:09:03.253723084Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4987648-mgmt-1\",\"clusterName\":\"ci01-j4987648-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 10:38	PR #6866	2107420154424987648

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j5137536-mgmt-2","clusterName":"ci01-j513753

Signal:

provision

none

Full failure examples (1)

time=2026-10-09T14:05:11.912Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-09T14:35:11.381Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-09T14:05:11.922891129Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-09T14:05:14.313036396Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-09T14:05:14.653688404Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-09 13:37	PR #6746	2108552522435137536

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-04T19:42:22

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.004s.
Expected success, but got an error:
    <*fmt.wrapError | 0x260bcea34260>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);
    let _endTime = datetime(2026-10-05T19:42:22.1821449Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-xj24lxzh8jt8'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-05T19:42:30.9508158Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4", ActivityId:"f7c0f10d-28d1-4f0c-aa3d-b966d3722931", SubActivityId:"f7c0f10d-28d1-4f0c-aa3d-b966d3722931", ActivityType:"GW.Http.CallContext", ParentActivityId:"f7c0f10d-28d1-4f0c-aa3d-b966d3722931", ActivityStack:"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x260bd06af728>{
            errs: [
                <*fmt.wrapError | 0x260bcea34240>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x260bcea34220>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x260bcfe34b60>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x260bcfe34b00>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x260bcef76000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.004s.
Expected success, but got an error:
    <*fmt.wrapError | 0x260bcea34260>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);
    let _endTime = datetime(2026-10-05T19:42:22.1821449Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-xj24lxzh8jt8'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-05T19:42:30.9508158Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4", ActivityId:"f7c0f10d-28d1-4f0c-aa3d-b966d3722931", SubActivityId:"f7c0f10d-28d1-4f0c-aa3d-b966d3722931", ActivityType:"GW.Http.CallContext", ParentActivityId:"f7c0f10d-28d1-4f0c-aa3d-b966d3722931", ActivityStack:"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x260bd06af728>{
            errs: [
                <*fmt.wrapError | 0x260bcea34240>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x260bcea34220>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x260bcfe34b60>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x260bcfe34b00>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-04T19:42:22.1821447Z);\nlet _endTime = datetime(2026-10-05T19:42:22.1821449Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-xj24lxzh8jt8'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-05T19:42:30.9508158Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4\", ActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", SubActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"f7c0f10d-28d1-4f0c-aa3d-b966d3722931\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7caca62d-ed6e-422a-9706-85e0c5cf40b4 ARID=f7c0f10d-28d1-4f0c-aa3d-b966d3722931 > GW.Http.CallContext/f7c0f10d-28d1-4f0c-aa3d-b966d3722931)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x260bcef76000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 17:59	PR #7105	2107168847067877376

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-05T14:32:08

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 603.891s.
Expected success, but got an error:
    <*fmt.wrapError | 0x31dad2fb8f20>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);
    let _endTime = datetime(2026-10-06T14:32:08.6055728Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7527gfpkwwfb'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-06T14:32:14.9016360Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715", ActivityId:"343d6fdf-e473-43f0-9637-7e55e6c53f36", SubActivityId:"343d6fdf-e473-43f0-9637-7e55e6c53f36", ActivityType:"GW.Http.CallContext", ParentActivityId:"343d6fdf-e473-43f0-9637-7e55e6c53f36", ActivityStack:"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x31dad3d26420>{
            errs: [
                <*fmt.wrapError | 0x31dad2fb8f00>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x31dad2fb8ee0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x31dad4339380>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x31dad4339340>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x31dad2c5e300>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 603.891s.
Expected success, but got an error:
    <*fmt.wrapError | 0x31dad2fb8f20>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);
    let _endTime = datetime(2026-10-06T14:32:08.6055728Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7527gfpkwwfb'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-06T14:32:14.9016360Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715", ActivityId:"343d6fdf-e473-43f0-9637-7e55e6c53f36", SubActivityId:"343d6fdf-e473-43f0-9637-7e55e6c53f36", ActivityType:"GW.Http.CallContext", ParentActivityId:"343d6fdf-e473-43f0-9637-7e55e6c53f36", ActivityStack:"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x31dad3d26420>{
            errs: [
                <*fmt.wrapError | 0x31dad2fb8f00>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x31dad2fb8ee0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x31dad4339380>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x31dad4339340>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-05T14:32:08.6055725Z);\nlet _endTime = datetime(2026-10-06T14:32:08.6055728Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7527gfpkwwfb'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-06T14:32:14.9016360Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715\", ActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", SubActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"343d6fdf-e473-43f0-9637-7e55e6c53f36\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ec4af9b9-774c-4c44-b0ce-feca4b2ac715 ARID=343d6fdf-e473-43f0-9637-7e55e6c53f36 > GW.Http.CallContext/343d6fdf-e473-43f0-9637-7e55e6c53f36)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x31dad2c5e300>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 13:04	PR #6866	2107456857781047296

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0x339d2c183220>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);
    let _endTime = datetime(2026-10-07T22:32:32.2954057Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-n8d6fgpt4qrq'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T22:32:43.6043688Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd", ActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", SubActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityType:"GW.Http.CallContext", ParentActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityStack:"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x339d2b3c97a0>{
            errs: [
                <*fmt.wrapError | 0x339d2c183200>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x339d2c1831e0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x339d2c1830c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x339d2c183080>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x339d2ae3d200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0x339d2c183220>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);
    let _endTime = datetime(2026-10-07T22:32:32.2954057Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-n8d6fgpt4qrq'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-07T22:32:43.6043688Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd", ActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", SubActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityType:"GW.Http.CallContext", ParentActivityId:"135bb6a2-6f7b-4d3f-a01b-32572cd9031c", ActivityStack:"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x339d2b3c97a0>{
            errs: [
                <*fmt.wrapError | 0x339d2c183200>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x339d2c1831e0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x339d2c1830c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x339d2c183080>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-06T22:32:32.2954055Z);\nlet _endTime = datetime(2026-10-07T22:32:32.2954057Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-n8d6fgpt4qrq'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-07T22:32:43.6043688Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd\", ActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", SubActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"135bb6a2-6f7b-4d3f-a01b-32572cd9031c\", ActivityStack:\"(Activity stack: CRID=KGC.execute;ba806fbf-42fd-4d1f-9a02-0f02e32bcdfd ARID=135bb6a2-6f7b-4d3f-a01b-32572cd9031c > GW.Http.CallContext/135bb6a2-6f7b-4d3f-a01b-32572cd9031c)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x339d2ae3d200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 21:02	PR #6954	2107939651057618944

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.006s.
Expected success, but got an error:
    <*fmt.wrapError | 0x11eea31d1d40>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);
    let _endTime = datetime(2026-10-08T00:23:30.3337686Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7nt846zks6zp'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T00:23:37.3079994Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d", ActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", SubActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityType:"GW.Http.CallContext", ParentActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityStack:"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x11eea2518d38>{
            errs: [
                <*fmt.wrapError | 0x11eea31d1d20>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x11eea31d1d00>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x11eea2cd05c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x11eea2cd0560>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x11eea1622000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.006s.
Expected success, but got an error:
    <*fmt.wrapError | 0x11eea31d1d40>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);
    let _endTime = datetime(2026-10-08T00:23:30.3337686Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7nt846zks6zp'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T00:23:37.3079994Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d", ActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", SubActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityType:"GW.Http.CallContext", ParentActivityId:"2dbee824-d49b-4a64-bb2b-a493f4fef7a1", ActivityStack:"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x11eea2518d38>{
            errs: [
                <*fmt.wrapError | 0x11eea31d1d20>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x11eea31d1d00>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x11eea2cd05c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x11eea2cd0560>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T00:23:30.3337685Z);\nlet _endTime = datetime(2026-10-08T00:23:30.3337686Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7nt846zks6zp'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T00:23:37.3079994Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d\", ActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", SubActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"2dbee824-d49b-4a64-bb2b-a493f4fef7a1\", ActivityStack:\"(Activity stack: CRID=KGC.execute;b78e7740-aeae-43b4-b8e3-a48c2ee1f56d ARID=2dbee824-d49b-4a64-bb2b-a493f4fef7a1 > GW.Http.CallContext/2dbee824-d49b-4a64-bb2b-a493f4fef7a1)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x11eea1622000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 23:00	PR #6928	2107969278706716672

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...

full failure pattern:

failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0x119e8d352a00>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);
    let _endTime = datetime(2026-10-08T14:29:33.7915314Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7dxbsk584tbw'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T14:29:42.5063608Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666", ActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", SubActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityType:"GW.Http.CallContext", ParentActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityStack:"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x119e8dac0f30>{
            errs: [
                <*fmt.wrapError | 0x119e8d3529e0>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x119e8d3529c0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x119e8d36b280>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x119e8d36b1a0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x119e8b843000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0x119e8d352a00>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);
    let _endTime = datetime(2026-10-08T14:29:33.7915314Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7dxbsk584tbw'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T14:29:42.5063608Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666", ActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", SubActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityType:"GW.Http.CallContext", ParentActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityStack:"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x119e8dac0f30>{
            errs: [
                <*fmt.wrapError | 0x119e8d3529e0>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x119e8d3529c0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x119e8d36b280>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x119e8d36b1a0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x119e8b843000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-08 13:01	PR #7387	2108181128597737472

failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline...

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-07T14:40:32.913Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-10-07T14:40:33.412Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157 global:map[imageOverrides:map[governance_policy_propagator:governance-policy-propagator-acm-216@sha256:deb737b19932d520553c902ee5e0738ebeaad4592e8aa9649c41fec6fc00b62e] registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-10-07T14:40:33.412Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:40:33.440Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 namespace=multicluster-engine
time=2026-10-07T14:40:33.491Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 dryRun=false
time=2026-10-07T14:40:33.491Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:40:33.534Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1
time=2026-10-07T14:51:01.640Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 err="failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-10-07T14:51:01.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-07 13:55	PR #7397	2107832305794945024

failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 11/15 context deadline exceeded

full failure pattern:

failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 11/15 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-06T02:50:37.115Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-10-06T02:50:37.513Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 values=map[enabled:true]
time=2026-10-06T02:50:37.513Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-06T02:50:37.822Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 namespace=kubelet-ds
time=2026-10-06T02:50:38.180Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 dryRun=false
time=2026-10-06T02:50:38.180Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-06T02:50:38.350Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-06T02:55:39.967Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 11/15\ncontext deadline exceeded"
time=2026-10-06T02:55:39.968Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 11/15\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/kubelet-ds (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-06 02:05	PR #7355	2107291152553086976

failed to roll out Helm release: unable to continue with install: could not get information about the resource Deployment "maestro-agent" in namespace "<namespace>": the server wa...

full failure pattern:

failed to roll out Helm release: unable to continue with install: could not get information about the resource Deployment "maestro-agent" in namespace "<namespace>": the server was unable to return a response in the time allotted, but may still be processing

Signal:

provision

none

Full failure examples (1)

time=2026-10-05T22:09:08.885Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-05T22:09:09.811Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:f8b6549e-d7f6-4ed0-9e41-251cd63c5293 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j9785984.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j9785984-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j9785984-1 secret:hcp-underlay-j9785984-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-05T22:09:09.811Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-05T22:09:25.359Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-05T22:09:40.945Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-05T22:09:40.945Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-05T22:09:41.070Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-05T22:11:59.108Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="unable to continue with install: could not get information about the resource Deployment \"maestro-agent\" in namespace \"maestro\": the server was unable to return a response in the time allotted, but may still be processing the request (get deployments.apps maestro-agent)"
time=2026-10-05T22:11:59.108Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: unable to continue with install: could not get information about the resource Deployment \"maestro-agent\" in namespace \"maestro\": the server was unable to return a response in the time allotted, but may still be processing the request (get deployments.apps maestro-agent)"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/management/deploy (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 21:05	PR #6820	2107215529679785984

tls: failed to verify certificate: x509: certificate signed by unknown authority

full failure pattern:

tls: failed to verify certificate: x509: certificate signed by unknown authority

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_serve_default_ingress_certificate_public.go:162]: sample app must return HTTP 200 over trusted HTTPS through public ingress
Unexpected error:
    <*fmt.wrapError | 0x3372404d8860>: 
    VerifyDefaultIngressCertificate timed out after 35m0.006s: default ingress route must serve trusted HTTPS: Get "https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        msg: "VerifyDefaultIngressCertificate timed out after 35m0.006s: default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
        err: <*fmt.wrapError | 0x3372404d86a0>{
            msg: "default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            err: <*url.Error | 0x337240d25200>{
                Op: "Get",
                URL: "https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                Err: <*tls.CertificateVerificationError | 0x337240d251d0>{
                    UnverifiedCertificates: [
                        {
                            Raw: [48, 130, 3, 150, 48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            RawTBSCertificate: [48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1],
                            RawSubject: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            RawIssuer: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            Signature: [147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            SignatureAlgorithm: 4,
                            PublicKeyAlgorithm: 1,
                            PublicKey: <*rsa.PublicKey | 0x337240007ac0>{
                                N: {
                                    neg: false,
                                    abs: [9337245268397936185, 5255789798456801510, 14154494615915577176, 13109335347058024190, 5072296307041548036, 10439277193874690340, 17833462028520810340, 16099347852796190717, 15302004063425798626, 7170477405477910213, 9971114008945693550, 13006318046232270092, 8600807242817419703, 6308103462559865749, 15463010847702061686, 16363344735281340381, 7111570420315025654, 6109672876023564938, 15424636460196878899, 7713335239787772476, 10387744396337381444, 5635081974936846311, 14718196154500586111, 14611695969280247348, 1443315009325149712, 11815840286537703567, 12683726052456297414, 292525798815078696, 515881726454289397, 10099284332814716175, 2649542967606308371, 13084258991057181117],
                                },
                                E: 65537,
                            },
                            Version: 3,
                            SerialNumber: {
                                neg: false,
                                abs: [11686061353655275548, 5126660660355678701],
                            },
                            Issuer: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            Subject: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            NotBefore: {wall: 0, ext: 63926798559, loc: nil},
                            NotAfter: {wall: 0, ext: 63942523959, loc: nil},
                            KeyUsage: 5,
                            Extensions: [
                                {
                                    Id: [2, 5, 29, 15],
                                    Critical: true,
                                    Value: [3, 2, 5, 160],
                                },
                                {Id: [2, 5, 29, 19], Critical: false, Value: [48, 0]},
                                {
                                    Id: [2, 5, 29, 37],
                                    Critical: false,
                                    Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2],
                                },
                                {
                                    Id: [2, 5, 29, 17],
                                    Critical: false,
                                    Value: "0;\x829*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                                },
                                {
                                    Id: [2, 5, 29, 35],
                                    Critical: false,
                                    Value: [48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                                {
                                    Id: [2, 5, 29, 14],
                                    Critical: false,
                                    Value: [4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                            ],
                            ExtraExtensions: nil,
                            UnhandledCriticalExtensions: nil,
                            ExtKeyUsage: [1, 2],
                            UnknownExtKeyUsage: nil,
                            BasicConstraintsValid: true,
                            IsCA: false,
                            MaxPathLen: -1,
                            MaxPathLenZero: false,
                            SubjectKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            AuthorityKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            OCSPServer: nil,
                            IssuingCertificateURL: nil,
                            DNSNames: [
                                "*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                            ],
                            EmailAddresses: nil,
                            IPAddresses: nil,
                            URIs: nil,
                            PermittedDNSDomainsCritical: false,
                            PermittedDNSDomains: nil,
                            ExcludedDNSDomains: nil,
                            PermittedIPRanges: nil,
                            ExcludedIPRanges: nil,
                            PermittedEmailAddresses: nil,
                            ExcludedEmailAddresses: nil,
                            PermittedURIDomains: nil,
                            ExcludedURIDomains: nil,
                            CRLDistributionPoints: nil,
                            PolicyIdentifiers: nil,
                            Policies: nil,
                            InhibitAnyPolicy: 0,
                            InhibitAnyPolicyZero: false,
                            InhibitPolicyMapping: 0,
                            InhibitPolicyMappingZero: false,
                            RequireExplicitPolicy: 0,
                            RequireExplicitPolicyZero: false,
                            PolicyMappings: nil,
                        },
                    ],
                    Err: <x509.UnknownAuthorityError>{
                        Cert: {
                            Raw: [48, 130, 3, 150, 48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            RawTBSCertificate: [48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1],
                            RawSubject: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            RawIssuer: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            Signature: [147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            SignatureAlgorithm: 4,
                            PublicKeyAlgorithm: 1,
                            PublicKey: <*rsa.PublicKey | 0x337240007ac0>{
                                N: {
                                    neg: false,
                                    abs: [9337245268397936185, 5255789798456801510, 14154494615915577176, 13109335347058024190, 5072296307041548036, 10439277193874690340, 17833462028520810340, 16099347852796190717, 15302004063425798626, 7170477405477910213, 9971114008945693550, 13006318046232270092, 8600807242817419703, 6308103462559865749, 15463010847702061686, 16363344735281340381, 7111570420315025654, 6109672876023564938, 15424636460196878899, 7713335239787772476, 10387744396337381444, 5635081974936846311, 14718196154500586111, 14611695969280247348, 1443315009325149712, 11815840286537703567, 12683726052456297414, 292525798815078696, 515881726454289397, 10099284332814716175, 2649542967606308371, 13084258991057181117],
                                },
                                E: 65537,
                            },
                            Version: 3,
                            SerialNumber: {
                                neg: false,
                                abs: [11686061353655275548, 5126660660355678701],
                            },
                            Issuer: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            Subject: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            NotBefore: {wall: 0, ext: 63926798559, loc: nil},
                            NotAfter: {wall: 0, ext: 63942523959, loc: nil},
                            KeyUsage: 5,
                            Extensions: [
                                {
                                    Id: [2, 5, 29, 15],
                                    Critical: true,
                                    Value: [3, 2, 5, 160],
                                },
                                {Id: [2, 5, 29, 19], Critical: false, Value: [48, 0]},
                                {
                                    Id: [2, 5, 29, 37],
                                    Critical: false,
                                    Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2],
                                },
                                {
                                    Id: [2, 5, 29, 17],
                                    Critical: false,
                                    Value: "0;\x829*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                                },
                                {
                                    Id: [2, 5, 29, 35],
                                    Critical: false,
                                    Value: [48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                                {
                                    Id: [2, 5, 29, 14],
                                    Critical: false,
                                    Value: [4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                            ],
                            ExtraExtensions: nil,
                            UnhandledCriticalExtensions: nil,
                            ExtKeyUsage: [1, 2],
                            UnknownExtKeyUsage: nil,
                            BasicConstraintsValid: true,
                            IsCA: false,
                            MaxPathLen: -1,
                            MaxPathLenZero: false,
                            SubjectKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            AuthorityKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            OCSPServer: nil,
                            IssuingCertificateURL: nil,
                            DNSNames: [
                                "*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                            ],
                            EmailAddresses: nil,
                            IPAddresses: nil,
                            URIs: nil,
                            PermittedDNSDomainsCritical: false,
                            PermittedDNSDomains: nil,
                            ExcludedDNSDomains: nil,
                            PermittedIPRanges: nil,
                            ExcludedIPRanges: nil,
                            PermittedEmailAddresses: nil,
                            ExcludedEmailAddresses: nil,
                            PermittedURIDomains: nil,
                            ExcludedURIDomains: nil,
                            CRLDistributionPoints: nil,
                            PolicyIdentifiers: nil,
                            Policies: nil,
                            InhibitAnyPolicy: 0,
                            InhibitAnyPolicyZero: false,
                            InhibitPolicyMapping: 0,
                            InhibitPolicyMappingZero: false,
                            RequireExplicitPolicy: 0,
                            RequireExplicitPolicyZero: false,
                            PolicyMappings: nil,
                        },
                        hintErr: nil,
                        hintCert: nil,
                    },
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_serve_default_ingress_certificate_public.go:162]: sample app must return HTTP 200 over trusted HTTPS through public ingress
Unexpected error:
    <*fmt.wrapError | 0x3372404d8860>: 
    VerifyDefaultIngressCertificate timed out after 35m0.006s: default ingress route must serve trusted HTTPS: Get "https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        msg: "VerifyDefaultIngressCertificate timed out after 35m0.006s: default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
        err: <*fmt.wrapError | 0x3372404d86a0>{
            msg: "default ingress route must serve trusted HTTPS: Get \"https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
            err: <*url.Error | 0x337240d25200>{
                Op: "Get",
                URL: "https://agnhost-e2e-sample-app-dlm4f.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                Err: <*tls.CertificateVerificationError | 0x337240d251d0>{
                    UnverifiedCertificates: [
                        {
                            Raw: [48, 130, 3, 150, 48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            RawTBSCertificate: [48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1],
                            RawSubject: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            RawIssuer: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            Signature: [147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            SignatureAlgorithm: 4,
                            PublicKeyAlgorithm: 1,
                            PublicKey: <*rsa.PublicKey | 0x337240007ac0>{
                                N: {
                                    neg: false,
                                    abs: [9337245268397936185, 5255789798456801510, 14154494615915577176, 13109335347058024190, 5072296307041548036, 10439277193874690340, 17833462028520810340, 16099347852796190717, 15302004063425798626, 7170477405477910213, 9971114008945693550, 13006318046232270092, 8600807242817419703, 6308103462559865749, 15463010847702061686, 16363344735281340381, 7111570420315025654, 6109672876023564938, 15424636460196878899, 7713335239787772476, 10387744396337381444, 5635081974936846311, 14718196154500586111, 14611695969280247348, 1443315009325149712, 11815840286537703567, 12683726052456297414, 292525798815078696, 515881726454289397, 10099284332814716175, 2649542967606308371, 13084258991057181117],
                                },
                                E: 65537,
                            },
                            Version: 3,
                            SerialNumber: {
                                neg: false,
                                abs: [11686061353655275548, 5126660660355678701],
                            },
                            Issuer: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            Subject: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            NotBefore: {wall: 0, ext: 63926798559, loc: nil},
                            NotAfter: {wall: 0, ext: 63942523959, loc: nil},
                            KeyUsage: 5,
                            Extensions: [
                                {
                                    Id: [2, 5, 29, 15],
                                    Critical: true,
                                    Value: [3, 2, 5, 160],
                                },
                                {Id: [2, 5, 29, 19], Critical: false, Value: [48, 0]},
                                {
                                    Id: [2, 5, 29, 37],
                                    Critical: false,
                                    Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2],
                                },
                                {
                                    Id: [2, 5, 29, 17],
                                    Critical: false,
                                    Value: "0;\x829*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                                },
                                {
                                    Id: [2, 5, 29, 35],
                                    Critical: false,
                                    Value: [48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                                {
                                    Id: [2, 5, 29, 14],
                                    Critical: false,
                                    Value: [4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                            ],
                            ExtraExtensions: nil,
                            UnhandledCriticalExtensions: nil,
                            ExtKeyUsage: [1, 2],
                            UnknownExtKeyUsage: nil,
                            BasicConstraintsValid: true,
                            IsCA: false,
                            MaxPathLen: -1,
                            MaxPathLenZero: false,
                            SubjectKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            AuthorityKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            OCSPServer: nil,
                            IssuingCertificateURL: nil,
                            DNSNames: [
                                "*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                            ],
                            EmailAddresses: nil,
                            IPAddresses: nil,
                            URIs: nil,
                            PermittedDNSDomainsCritical: false,
                            PermittedDNSDomains: nil,
                            ExcludedDNSDomains: nil,
                            PermittedIPRanges: nil,
                            ExcludedIPRanges: nil,
                            PermittedEmailAddresses: nil,
                            ExcludedEmailAddresses: nil,
                            PermittedURIDomains: nil,
                            ExcludedURIDomains: nil,
                            CRLDistributionPoints: nil,
                            PolicyIdentifiers: nil,
                            Policies: nil,
                            InhibitAnyPolicy: 0,
                            InhibitAnyPolicyZero: false,
                            InhibitPolicyMapping: 0,
                            InhibitPolicyMappingZero: false,
                            RequireExplicitPolicy: 0,
                            RequireExplicitPolicyZero: false,
                            PolicyMappings: nil,
                        },
                    ],
                    Err: <x509.UnknownAuthorityError>{
                        Cert: {
                            Raw: [48, 130, 3, 150, 48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            RawTBSCertificate: [48, 130, 2, 126, 160, 3, 2, 1, 2, 2, 16, 71, 37, 142, 181, 88, 120, 65, 237, 162, 45, 58, 226, 192, 11, 16, 28, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 53, 49, 50, 48, 50, 51, 57, 90, 23, 13, 50, 55, 48, 52, 48, 53, 49, 50, 49, 50, 51, 57, 90, 48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1, 163, 129, 195, 48, 129, 192, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 68, 6, 3, 85, 29, 17, 4, 61, 48, 59, 130, 57, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 98, 48, 106, 56, 114, 48, 104, 55, 108, 56, 113, 53, 116, 51, 107, 46, 48, 109, 118, 116, 46, 106, 54, 55, 53, 50, 55, 54, 56, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 181, 148, 160, 58, 159, 44, 153, 189, 36, 197, 17, 74, 137, 78, 238, 19, 140, 39, 221, 151, 167, 22, 205, 15, 7, 40, 199, 178, 107, 108, 83, 245, 4, 15, 66, 170, 6, 2, 153, 40, 176, 5, 165, 164, 185, 6, 115, 198, 163, 250, 76, 35, 57, 212, 96, 143, 20, 7, 175, 90, 157, 208, 174, 16, 202, 199, 44, 17, 102, 244, 198, 52, 204, 65, 137, 108, 19, 135, 26, 127, 78, 51, 213, 51, 68, 81, 63, 231, 144, 40, 174, 125, 233, 62, 128, 68, 107, 11, 69, 136, 153, 13, 122, 60, 214, 15, 81, 35, 39, 146, 218, 51, 84, 201, 235, 28, 61, 57, 74, 138, 98, 177, 95, 158, 90, 13, 112, 246, 227, 22, 71, 87, 11, 179, 223, 221, 214, 151, 166, 112, 127, 75, 18, 118, 87, 138, 226, 173, 108, 74, 219, 149, 119, 92, 52, 158, 73, 189, 57, 183, 180, 127, 185, 90, 69, 109, 85, 12, 138, 96, 131, 92, 166, 33, 19, 110, 99, 130, 167, 53, 159, 121, 182, 197, 212, 91, 163, 159, 72, 20, 33, 226, 223, 108, 95, 146, 249, 93, 95, 253, 247, 125, 47, 58, 173, 102, 103, 100, 144, 223, 195, 76, 84, 188, 209, 36, 70, 100, 106, 157, 248, 188, 183, 4, 181, 237, 183, 10, 186, 221, 158, 254, 196, 110, 221, 220, 4, 211, 191, 88, 72, 240, 80, 249, 175, 35, 184, 230, 129, 148, 143, 25, 203, 142, 78, 57, 2, 3, 1, 0, 1],
                            RawSubject: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            RawIssuer: [48, 36, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 114, 101, 115, 101, 114, 118, 101, 100, 46, 104, 99, 112, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                            Signature: [147, 134, 1, 10, 99, 115, 212, 128, 154, 247, 88, 143, 132, 18, 62, 104, 53, 29, 163, 204, 160, 86, 14, 144, 189, 8, 184, 211, 37, 11, 40, 146, 196, 120, 87, 249, 82, 39, 175, 243, 133, 128, 16, 181, 237, 205, 54, 58, 227, 14, 179, 200, 144, 127, 45, 145, 129, 24, 170, 240, 41, 228, 61, 99, 51, 77, 144, 12, 29, 169, 159, 25, 218, 17, 199, 182, 75, 152, 126, 179, 115, 15, 107, 196, 109, 85, 95, 213, 213, 58, 180, 53, 66, 153, 230, 39, 58, 144, 242, 96, 16, 226, 207, 0, 92, 163, 55, 242, 81, 244, 35, 243, 253, 19, 247, 198, 16, 11, 52, 54, 149, 220, 217, 58, 119, 161, 173, 86, 78, 177, 249, 121, 221, 175, 160, 140, 40, 7, 212, 180, 234, 200, 20, 32, 223, 73, 100, 92, 83, 139, 213, 173, 70, 18, 179, 17, 246, 69, 185, 16, 112, 42, 163, 187, 68, 169, 236, 189, 25, 89, 253, 108, 72, 149, 151, 150, 250, 244, 61, 80, 104, 174, 22, 117, 197, 243, 100, 97, 162, 170, 161, 255, 29, 154, 36, 83, 33, 119, 148, 212, 184, 52, 75, 236, 171, 107, 201, 4, 183, 186, 61, 72, 200, 2, 92, 39, 159, 16, 28, 82, 202, 52, 157, 129, 245, 113, 22, 70, 206, 100, 38, 180, 159, 164, 96, 252, 63, 248, 44, 185, 36, 172, 178, 93, 36, 254, 17, 239, 118, 89, 117, 197, 232, 50, 216, 210],
                            SignatureAlgorithm: 4,
                            PublicKeyAlgorithm: 1,
                            PublicKey: <*rsa.PublicKey | 0x337240007ac0>{
                                N: {
                                    neg: false,
                                    abs: [9337245268397936185, 5255789798456801510, 14154494615915577176, 13109335347058024190, 5072296307041548036, 10439277193874690340, 17833462028520810340, 16099347852796190717, 15302004063425798626, 7170477405477910213, 9971114008945693550, 13006318046232270092, 8600807242817419703, 6308103462559865749, 15463010847702061686, 16363344735281340381, 7111570420315025654, 6109672876023564938, 15424636460196878899, 7713335239787772476, 10387744396337381444, 5635081974936846311, 14718196154500586111, 14611695969280247348, 1443315009325149712, 11815840286537703567, 12683726052456297414, 292525798815078696, 515881726454289397, 10099284332814716175, 2649542967606308371, 13084258991057181117],
                                },
                                E: 65537,
                            },
                            Version: 3,
                            SerialNumber: {
                                neg: false,
                                abs: [11686061353655275548, 5126660660355678701],
                            },
                            Issuer: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            Subject: {
                                Country: nil,
                                Organization: nil,
                                OrganizationalUnit: nil,
                                Locality: nil,
                                Province: nil,
                                StreetAddress: nil,
                                PostalCode: nil,
                                SerialNumber: "",
                                CommonName: "reserved.hcp.osadev.cloud",
                                Names: [
                                    {
                                        Type: [..., ..., ..., ...],
                                        Value: <string>"reserved.hcp.osadev.cloud",
                                    },
                                ],
                                ExtraNames: nil,
                            },
                            NotBefore: {wall: 0, ext: 63926798559, loc: nil},
                            NotAfter: {wall: 0, ext: 63942523959, loc: nil},
                            KeyUsage: 5,
                            Extensions: [
                                {
                                    Id: [2, 5, 29, 15],
                                    Critical: true,
                                    Value: [3, 2, 5, 160],
                                },
                                {Id: [2, 5, 29, 19], Critical: false, Value: [48, 0]},
                                {
                                    Id: [2, 5, 29, 37],
                                    Critical: false,
                                    Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2],
                                },
                                {
                                    Id: [2, 5, 29, 17],
                                    Critical: false,
                                    Value: "0;\x829*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                                },
                                {
                                    Id: [2, 5, 29, 35],
                                    Critical: false,
                                    Value: [48, 22, 128, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                                {
                                    Id: [2, 5, 29, 14],
                                    Critical: false,
                                    Value: [4, 20, 168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                                },
                            ],
                            ExtraExtensions: nil,
                            UnhandledCriticalExtensions: nil,
                            ExtKeyUsage: [1, 2],
                            UnknownExtKeyUsage: nil,
                            BasicConstraintsValid: true,
                            IsCA: false,
                            MaxPathLen: -1,
                            MaxPathLenZero: false,
                            SubjectKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            AuthorityKeyId: [168, 251, 204, 255, 251, 89, 21, 98, 0, 191, 78, 208, 215, 20, 131, 219, 93, 195, 226, 223],
                            OCSPServer: nil,
                            IssuingCertificateURL: nil,
                            DNSNames: [
                                "*.apps.aro.b0j8r0h7l8q5t3k.0mvt.j6752768.hcp.osadev.cloud",
                            ],
                            EmailAddresses: nil,
                            IPAddresses: nil,
                            URIs: nil,
                            PermittedDNSDomainsCritical: false,
                            PermittedDNSDomains: nil,
                            ExcludedDNSDomains: nil,
                            PermittedIPRanges: nil,
                            ExcludedIPRanges: nil,
                            PermittedEmailAddresses: nil,
                            ExcludedEmailAddresses: nil,
                            PermittedURIDomains: nil,
                            ExcludedURIDomains: nil,
                            CRLDistributionPoints: nil,
                            PolicyIdentifiers: nil,
                            Policies: nil,
                            InhibitAnyPolicy: 0,
                            InhibitAnyPolicyZero: false,
                            InhibitPolicyMapping: 0,
                            InhibitPolicyMappingZero: false,
                            RequireExplicitPolicy: 0,
                            RequireExplicitPolicyZero: false,
                            PolicyMappings: nil,
                        },
                        hintErr: nil,
                        hintCert: nil,
                    },
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate through public ingress with OCP candidate channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-05 11:10	PR #7325	2107065983326752768

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

107

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x23a798e4e3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7kr9hzf26ffd" to finish creating: GET https://rp.j0742656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357",
      "name": "2a131d64-b48f-453b-b60d-f4063709a357",
      "status": "Failed",
      "startTime": "2026-10-10T03:18:06.524337179Z",
      "endTime": "2026-10-10T03:47:07.577893958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x23a798e4e3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7kr9hzf26ffd" to finish creating: GET https://rp.j0742656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357",
      "name": "2a131d64-b48f-453b-b60d-f4063709a357",
      "status": "Failed",
      "startTime": "2026-10-10T03:18:06.524337179Z",
      "endTime": "2026-10-10T03:47:07.577893958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x560689fa080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-jzlxwvqzg694" to finish creating: GET https://rp.j5842176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "name": "efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "status": "Failed",
      "startTime": "2026-10-10T02:25:01.899620825Z",
      "endTime": "2026-10-10T02:54:06.245842706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x560689fa080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-jzlxwvqzg694" to finish creating: GET https://rp.j5842176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "name": "efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "status": "Failed",
      "startTime": "2026-10-10T02:25:01.899620825Z",
      "endTime": "2026-10-10T02:54:06.245842706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (37)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	67
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	63
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	62
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	62
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	59
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	57
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	56
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	55
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	54
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	46
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	22
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	18
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with a private key vault using an available API version	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.21 minor to 4.22 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate through public ingress with OCP candidate channel for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a fully private cluster with private KAS and private ingress and verify both are only reachable from VNet	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a fully private cluster with private KAS, private ingress, and private KeyVault and verify all are operational	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (107)

Date (UTC)	Associated PR	Prow job
2026-10-10 02:02	PR #7439	2108739904140742656
2026-10-10 01:21	PR #7177	2108729709905842176
2026-10-09 18:00	PR #7427	2108618645423788032
2026-10-09 12:43	PR #7401	2108538835301306368
2026-10-09 11:01	PR #7397	2108513285430054912
2026-10-09 08:49	PR #6944	2108480044224483328
2026-10-09 07:02	PR #7397	2108452968792592384
2026-10-09 03:43	PR #7438	2108402844888469504
2026-10-09 02:03	PR #7364	2108377905737240576
2026-10-08 18:12	PR #6364	2108259306926247936
2026-10-08 18:00	PR #7387	2108256216495951872
2026-10-08 15:37	PR #7431	2108220296300335104
2026-10-08 14:53	PR #7361	2108209233152446464
2026-10-08 12:00	PR #7389	2108165661279653888
2026-10-08 11:08	PR #6364	2108152449343688704
2026-10-08 09:37	PR #7158	2108129603208876032
2026-10-08 09:09	PR #7292	2108122723405271040
2026-10-08 08:16	PR #7389	2108109354774302720
2026-10-08 07:10	PR #7412	2108092712069632000
2026-10-08 03:54	PR #7177	2108043334072143872
2026-10-07 21:02	PR #6954	2107939651057618944
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 10:22	PR #6688	2107778619966230528
2026-10-07 09:31	PR #7405	2107765914286952448
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 07:47	PR #7310	2107739528360890368
2026-10-07 03:55	PR #7397	2107681070840287232
2026-10-06 21:46	PR #7374	2107588360917749760
2026-10-06 21:46	PR #7389	2107588315996753920
2026-10-06 21:35	PR #7143	2107585481037320192
2026-10-06 21:16	PR #6364	2107580707109867520
2026-10-06 18:48	PR #6866	2107543602224697344
2026-10-06 18:48	PR #7374	2107543570377347072
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 18:48	PR #7392	2107543477116997632
2026-10-06 16:04	PR #6191	2107502219598237696
2026-10-06 15:05	PR #7288	2107487876705947648
2026-10-06 15:03	PR #7313	2107486936871473152
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 14:05	PR #6688	2107472359333040128
2026-10-06 12:27	PR #7383	2107447564306485248
2026-10-06 12:18	PR #7162	2107445319187828736
2026-10-06 11:53	PR #7288	2107439009616629760
2026-10-06 11:38	PR #7378	2107435326405349376
2026-10-06 11:02	PR #6688	2107426149817651200
2026-10-06 10:37	PR #7342	2107419847431294976
2026-10-06 10:36	PR #6191	2107419692363681792
2026-10-06 10:32	PR #7377	2107418802831495168
2026-10-06 10:00	PR #7374	2107410727168905216
2026-10-06 08:09	PR #6746	2107382803778768896
2026-10-06 07:37	PR #6364	2107374862493290496
2026-10-06 06:56	PR #7331	2107364414918234112
2026-10-06 05:43	PR #7310	2107345968763506688
2026-10-06 04:04	PR #7105	2107321129289912320
2026-10-06 02:37	PR #6364	2107299133835448320
2026-10-06 02:05	PR #7351	2107291222019149824
2026-10-06 02:05	PR #7279	2107291164120977408
2026-10-06 01:48	PR #7105	2107286743806906368
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-06 00:22	PR #7350	2107265250691649536
2026-10-05 23:05	PR #7365	2107245789225947136
2026-10-05 23:05	PR #7355	2107245759840653312
2026-10-05 23:05	PR #7356	2107245750499938304
2026-10-05 22:59	PR #7331	2107244242656038912
2026-10-05 22:00	PR #7279	2107229401274912768
2026-10-05 21:04	PR #7366	2107215519332438016
2026-10-05 21:04	PR #7362	2107215468136763392
2026-10-05 20:28	PR #7319	2107206250759458816
2026-10-05 19:55	PR #7365	2107198083426160640
2026-10-05 19:49	PR #7332	2107196586068021248
2026-10-05 19:39	PR #7349	2107193894448926720
2026-10-05 19:39	PR #7278	2107193887125671936
2026-10-05 18:56	PR #7363	2107183062990196736
2026-10-05 17:59	PR #7342	2107168939829104640
2026-10-05 17:59	PR #7351	2107168887228338176
2026-10-05 17:59	PR #7352	2107168871638110208
2026-10-05 17:59	PR #7355	2107168836758278144
2026-10-05 17:08	PR #7325	2107155963554304000
2026-10-05 16:34	PR #7362	2107147472735309824
2026-10-05 16:21	PR #7088	2107144096983289856
2026-10-05 15:32	PR #7279	2107131932365033472
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 14:44	PR #6746	2107119792417673216
2026-10-05 14:36	PR #7359	2107117812953649152
2026-10-05 14:36	PR #6944	2107117611937435648
2026-10-05 13:11	PR #7158	2107096289194807296
2026-10-05 12:56	PR #7313	2107092599713042432
2026-10-05 12:11	PR #7355	2107081303852584960
2026-10-05 11:14	PR #7351	2107066931902156800
2026-10-05 11:11	PR #7349	2107066190932217856
2026-10-05 11:11	PR #7292	2107066130689429504
2026-10-05 10:45	PR #6889	2107059552468865024
2026-10-05 07:24	PR #7215	2107008962619510784
2026-10-05 06:58	PR #7342	2107002394956009472
2026-10-05 06:51	PR #7331	2107000790592786432
2026-10-05 06:50	PR #7105	2107000443698679808
2026-10-05 04:56	PR #7345	2106971795000135680

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T18:33:37Z
  Ended:       2026-10-11T18:42:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8107776-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T15:52:40Z
  Ended:       2026-10-11T16:12:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1984128-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T21:28:57Z
  Ended:       2026-10-09T21:36:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8166656-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	95

Affected runs (95)

Date (UTC)	Associated PR	Prow job
2026-10-11 17:11	PR #7457	2109331063628107776
2026-10-11 14:35	PR #7457	2109291724151984128
2026-10-09 20:00	PR #7439	2108648873588166656
2026-10-09 14:46	PR #7444	2108569711095582720
2026-10-09 07:02	PR #7397	2108452968792592384
2026-10-09 01:49	PR #7437	2108374328641851392
2026-10-08 22:02	PR #7389	2108317197410504704
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 18:00	PR #7414	2108256251258343424
2026-10-08 18:00	PR #7389	2108256213476052992
2026-10-08 16:13	PR #7232	2108229382379147264
2026-10-08 12:00	PR #7389	2108165661279653888
2026-10-08 11:50	PR #7401	2108163022403932160
2026-10-08 11:12	PR #7383	2108153534082977792
2026-10-08 11:09	PR #7364	2108152878592954368
2026-10-08 11:08	PR #6364	2108152449343688704
2026-10-08 09:41	PR #7288	2108130674367008768
2026-10-08 09:33	PR #7387	2108128762271895552
2026-10-08 09:12	PR #7420	2108123255498870784
2026-10-08 08:20	PR #7414	2108110245078241280
2026-10-08 04:44	PR #7414	2108055963964542976
2026-10-08 03:54	PR #7177	2108043334072143872
2026-10-08 03:07	PR #7412	2108031501605015552
2026-10-08 02:14	PR #7325	2108018145892503552
2026-10-08 02:04	PR #7232	2108015628659986432
2026-10-07 22:55	PR #7417	2107967978476670976
2026-10-07 22:42	PR #7414	2107964840596213760
2026-10-07 20:10	PR #6364	2107926490833227776
2026-10-07 20:09	PR #7305	2107926353130033152
2026-10-07 19:30	PR #7414	2107916465603612672
2026-10-07 19:20	PR #7412	2107914072107257856
2026-10-07 18:41	PR #7288	2107904235558408192
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:48	PR #7322	2107875761300967424
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:37	PR #7310	2107872947002675200
2026-10-07 14:42	PR #7410	2107844183950626816
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 11:55	PR #7405	2107802095104036864
2026-10-07 10:22	PR #6688	2107778619966230528
2026-10-07 08:38	PR #7133	2107752436901875712
2026-10-07 08:36	PR #7403	2107751899905134592
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7143	2107750854802018304
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 08:29	PR #7370	2107750168211230720
2026-10-07 08:29	PR #6968	2107750153682161664
2026-10-07 07:47	PR #7310	2107739528360890368
2026-10-07 03:55	PR #7397	2107681070840287232
2026-10-07 03:51	PR #7097	2107680144293367808
2026-10-06 21:46	PR #7374	2107588360917749760
2026-10-06 20:30	PR #6775	2107569137193586688
2026-10-06 18:48	PR #7388	2107543517562671104
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 18:48	PR #7392	2107543477116997632
2026-10-06 15:53	PR #7389	2107499536325808128
2026-10-06 15:30	PR #7292	2107493772093820928
2026-10-06 15:05	PR #7288	2107487876705947648
2026-10-06 15:03	PR #7313	2107486936871473152
2026-10-06 12:18	PR #7162	2107445319187828736
2026-10-06 11:53	PR #7288	2107439009616629760
2026-10-06 11:02	PR #6688	2107426149817651200
2026-10-06 03:45	PR #6746	2107316415294869504
2026-10-06 01:48	PR #7105	2107286743806906368
2026-10-06 00:02	PR #7362	2107260288968429568
2026-10-05 23:05	PR #7313	2107245793541885952
2026-10-05 23:05	PR #7365	2107245789225947136
2026-10-05 23:05	PR #7342	2107245776429125632
2026-10-05 23:05	PR #7356	2107245750499938304
2026-10-05 22:03	PR #7143	2107230106127699968
2026-10-05 21:45	PR #7350	2107225642733932544
2026-10-05 21:07	PR #6364	2107216083751538688
2026-10-05 19:39	PR #7349	2107193894448926720
2026-10-05 19:04	PR #7279	2107185267151474688
2026-10-05 18:56	PR #7363	2107183062990196736
2026-10-05 17:59	PR #7342	2107168939829104640
2026-10-05 17:59	PR #7352	2107168871638110208
2026-10-05 16:56	PR #7305	2107152898931560448
2026-10-05 16:07	PR #7310	2107140685202198528
2026-10-05 15:17	PR #7143	2107127981024481280
2026-10-05 14:44	PR #6746	2107119792417673216
2026-10-05 14:36	PR #6944	2107117611937435648
2026-10-05 13:11	PR #7158	2107096289194807296
2026-10-05 12:56	PR #7313	2107092599713042432
2026-10-05 12:14	PR #7356	2107081906662150144
2026-10-05 12:11	PR #7355	2107081303852584960
2026-10-05 11:33	PR #7348	2107071771768262656
2026-10-05 11:14	PR #7351	2107066931902156800
2026-10-05 11:14	PR #7352	2107066918295834624
2026-10-05 11:11	PR #7349	2107066190932217856
2026-10-05 11:10	PR #7325	2107065983326752768

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xe1c86824ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-cnc5f6qgdgz8" to finish creating: GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932",
      "name": "58fb2269-9049-4fd2-9274-ac2190a19932",
      "status": "Failed",
      "startTime": "2026-10-11T12:57:29.816928878Z",
      "endTime": "2026-10-11T13:26:38.515811706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xr7ss-282dw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xr7ss-282dw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
            correlation ID: b9b2478c-6a44-4ebb-ade0-4de59279ccb4
            client request ID: 4eee6950-aeb4-430b-b956-66f798ff6c28
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xe1c86824ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-cnc5f6qgdgz8" to finish creating: GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932",
      "name": "58fb2269-9049-4fd2-9274-ac2190a19932",
      "status": "Failed",
      "startTime": "2026-10-11T12:57:29.816928878Z",
      "endTime": "2026-10-11T13:26:38.515811706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xr7ss-282dw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xr7ss-282dw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
            correlation ID: b9b2478c-6a44-4ebb-ade0-4de59279ccb4
            client request ID: 4eee6950-aeb4-430b-b956-66f798ff6c28

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x7f83ee01c00>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c4ck5gxmvgl2" to finish creating: GET https://rp.j2487424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "name": "b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "status": "Failed",
      "startTime": "2026-10-10T01:21:08.093175992Z",
      "endTime": "2026-10-10T01:50:13.893881459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x7f83ee01c00>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c4ck5gxmvgl2" to finish creating: GET https://rp.j2487424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "name": "b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "status": "Failed",
      "startTime": "2026-10-10T01:21:08.093175992Z",
      "endTime": "2026-10-10T01:50:13.893881459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19485d7e23a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-h78g452l2k2m" to finish creating: GET https://rp.j1850240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c",
      "name": "2445641e-62af-46ed-8294-3ebf3cab568c",
      "status": "Failed",
      "startTime": "2026-10-09T03:36:09.491861222Z",
      "endTime": "2026-10-09T04:05:13.503264505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19485d7e23a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-h78g452l2k2m" to finish creating: GET https://rp.j1850240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c",
      "name": "2445641e-62af-46ed-8294-3ebf3cab568c",
      "status": "Failed",
      "startTime": "2026-10-09T03:36:09.491861222Z",
      "endTime": "2026-10-09T04:05:13.503264505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	78
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	2

Affected runs (79)

Date (UTC)	Associated PR	Prow job
2026-10-11 12:00	PR #7219	2109252833235701760
2026-10-10 00:06	PR #7165	2108710710912487424
2026-10-09 02:04	PR #7292	2108377922371850240
2026-10-08 14:06	PR #7422	2108197422965460992
2026-10-07 19:32	PR #7325	2107916955439599616
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 09:31	PR #7405	2107765914286952448
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 03:55	PR #7397	2107681070840287232
2026-10-06 21:46	PR #7374	2107588360917749760
2026-10-06 21:46	PR #7389	2107588315996753920
2026-10-06 21:35	PR #7143	2107585481037320192
2026-10-06 18:48	PR #7374	2107543570377347072
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 18:48	PR #7392	2107543477116997632
2026-10-06 16:04	PR #6191	2107502219598237696
2026-10-06 15:05	PR #7288	2107487876705947648
2026-10-06 15:03	PR #7313	2107486936871473152
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 14:05	PR #6688	2107472359333040128
2026-10-06 12:27	PR #7383	2107447564306485248
2026-10-06 12:18	PR #7162	2107445319187828736
2026-10-06 11:53	PR #7288	2107439009616629760
2026-10-06 11:38	PR #7378	2107435326405349376
2026-10-06 11:02	PR #6688	2107426149817651200
2026-10-06 10:37	PR #7342	2107419847431294976
2026-10-06 10:36	PR #6191	2107419692363681792
2026-10-06 10:32	PR #7377	2107418802831495168
2026-10-06 08:09	PR #6746	2107382803778768896
2026-10-06 07:37	PR #6364	2107374862493290496
2026-10-06 05:43	PR #7310	2107345968763506688
2026-10-06 04:04	PR #7105	2107321129289912320
2026-10-06 02:37	PR #6364	2107299133835448320
2026-10-06 02:05	PR #7351	2107291222019149824
2026-10-06 01:48	PR #7105	2107286743806906368
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-05 23:05	PR #7365	2107245789225947136
2026-10-05 23:05	PR #7355	2107245759840653312
2026-10-05 23:05	PR #7356	2107245750499938304
2026-10-05 22:59	PR #7331	2107244242656038912
2026-10-05 22:00	PR #7279	2107229401274912768
2026-10-05 21:04	PR #7362	2107215468136763392
2026-10-05 20:28	PR #7319	2107206250759458816
2026-10-05 19:55	PR #7365	2107198083426160640
2026-10-05 19:49	PR #7332	2107196586068021248
2026-10-05 19:39	PR #7349	2107193894448926720
2026-10-05 19:39	PR #7278	2107193887125671936
2026-10-05 18:56	PR #7363	2107183062990196736
2026-10-05 17:59	PR #7342	2107168939829104640
2026-10-05 17:59	PR #7351	2107168887228338176
2026-10-05 17:59	PR #7352	2107168871638110208
2026-10-05 17:59	PR #7355	2107168836758278144
2026-10-05 17:08	PR #7325	2107155963554304000
2026-10-05 16:34	PR #7362	2107147472735309824
2026-10-05 16:21	PR #7088	2107144096983289856
2026-10-05 15:32	PR #7279	2107131932365033472
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 14:44	PR #6746	2107119792417673216
2026-10-05 14:36	PR #7359	2107117812953649152
2026-10-05 14:36	PR #6944	2107117611937435648
2026-10-05 13:11	PR #7158	2107096289194807296
2026-10-05 12:56	PR #7313	2107092599713042432
2026-10-05 12:11	PR #7355	2107081303852584960
2026-10-05 11:33	PR #7348	2107071771768262656
2026-10-05 11:14	PR #7351	2107066931902156800
2026-10-05 07:24	PR #7215	2107008962619510784
2026-10-05 06:58	PR #7342	2107002394956009472
2026-10-05 06:51	PR #7331	2107000790592786432
2026-10-05 06:50	PR #7105	2107000443698679808
2026-10-05 04:56	PR #7345	2106971795000135680

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	48
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	41
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Verify cluster operators should provision an HCP cluster and verify the cluster operators	1

Affected runs (69)

Date (UTC)	Associated PR	Prow job
2026-10-09 02:04	PR #7292	2108377922371850240
2026-10-09 01:49	PR #7437	2108374328641851392
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 09:31	PR #7405	2107765914286952448
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-07 03:55	PR #7397	2107681070840287232
2026-10-06 21:46	PR #7374	2107588360917749760
2026-10-06 21:46	PR #7389	2107588315996753920
2026-10-06 21:35	PR #7143	2107585481037320192
2026-10-06 18:48	PR #7374	2107543570377347072
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 18:48	PR #7392	2107543477116997632
2026-10-06 16:04	PR #6191	2107502219598237696
2026-10-06 15:05	PR #7288	2107487876705947648
2026-10-06 15:03	PR #7313	2107486936871473152
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 14:05	PR #6688	2107472359333040128
2026-10-06 12:27	PR #7383	2107447564306485248
2026-10-06 12:18	PR #7162	2107445319187828736
2026-10-06 11:53	PR #7288	2107439009616629760
2026-10-06 11:38	PR #7378	2107435326405349376
2026-10-06 11:02	PR #6688	2107426149817651200
2026-10-06 10:37	PR #7342	2107419847431294976
2026-10-06 10:36	PR #6191	2107419692363681792
2026-10-06 10:32	PR #7377	2107418802831495168
2026-10-06 08:09	PR #6746	2107382803778768896
2026-10-06 07:37	PR #6364	2107374862493290496
2026-10-06 07:30	PR #7373	2107373015527329792
2026-10-06 07:13	PR #7073	2107368569695113216
2026-10-06 04:04	PR #7105	2107321129289912320
2026-10-06 02:37	PR #6364	2107299133835448320
2026-10-06 02:05	PR #7351	2107291222019149824
2026-10-06 01:48	PR #7105	2107286743806906368
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-05 23:05	PR #7355	2107245759840653312
2026-10-05 23:05	PR #7356	2107245750499938304
2026-10-05 22:00	PR #7279	2107229401274912768
2026-10-05 20:28	PR #7319	2107206250759458816
2026-10-05 19:55	PR #7365	2107198083426160640
2026-10-05 19:49	PR #7332	2107196586068021248
2026-10-05 19:39	PR #7349	2107193894448926720
2026-10-05 18:56	PR #7363	2107183062990196736
2026-10-05 17:59	PR #7342	2107168939829104640
2026-10-05 17:59	PR #7351	2107168887228338176
2026-10-05 17:59	PR #7352	2107168871638110208
2026-10-05 17:59	PR #7355	2107168836758278144
2026-10-05 17:08	PR #7325	2107155963554304000
2026-10-05 16:34	PR #7362	2107147472735309824
2026-10-05 16:21	PR #7088	2107144096983289856
2026-10-05 15:32	PR #7279	2107131932365033472
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 14:44	PR #6746	2107119792417673216
2026-10-05 14:36	PR #7359	2107117812953649152
2026-10-05 13:11	PR #7158	2107096289194807296
2026-10-05 12:56	PR #7313	2107092599713042432
2026-10-05 12:11	PR #7355	2107081303852584960
2026-10-05 11:14	PR #7351	2107066931902156800
2026-10-05 07:24	PR #7215	2107008962619510784
2026-10-05 06:58	PR #7342	2107002394956009472
2026-10-05 06:50	PR #7105	2107000443698679808
2026-10-05 04:56	PR #7345	2106971795000135680

ERROR CODE: DeadlineExceeded; provider Microsoft.Network

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (15)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	27
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	17
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	16
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	15
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	15
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	14
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	13
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	12
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	11
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	6
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	6
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (66)

Date (UTC)	Associated PR	Prow job
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:47	PR #7400	2107875420446658560
2026-10-07 16:41	PR #7412	2107873993380859904
2026-10-07 14:03	PR #7404	2107834229550223360
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 08:32	PR #7399	2107750792218808320
2026-10-07 08:31	PR #7371	2107750686056779776
2026-10-06 21:46	PR #7374	2107588360917749760
2026-10-06 21:35	PR #7143	2107585481037320192
2026-10-06 18:48	PR #7374	2107543570377347072
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 18:48	PR #7392	2107543477116997632
2026-10-06 16:04	PR #6191	2107502219598237696
2026-10-06 15:05	PR #7288	2107487876705947648
2026-10-06 15:03	PR #7313	2107486936871473152
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 14:05	PR #6688	2107472359333040128
2026-10-06 12:27	PR #7383	2107447564306485248
2026-10-06 12:18	PR #7162	2107445319187828736
2026-10-06 11:53	PR #7288	2107439009616629760
2026-10-06 11:38	PR #7378	2107435326405349376
2026-10-06 11:02	PR #6688	2107426149817651200
2026-10-06 10:37	PR #7342	2107419847431294976
2026-10-06 10:36	PR #6191	2107419692363681792
2026-10-06 10:32	PR #7377	2107418802831495168
2026-10-06 08:09	PR #6746	2107382803778768896
2026-10-06 07:37	PR #6364	2107374862493290496
2026-10-06 05:43	PR #7310	2107345968763506688
2026-10-06 04:04	PR #7105	2107321129289912320
2026-10-06 02:37	PR #6364	2107299133835448320
2026-10-06 02:05	PR #7351	2107291222019149824
2026-10-06 01:48	PR #7105	2107286743806906368
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-05 23:05	PR #7365	2107245789225947136
2026-10-05 23:05	PR #7355	2107245759840653312
2026-10-05 23:05	PR #7356	2107245750499938304
2026-10-05 22:59	PR #7331	2107244242656038912
2026-10-05 22:00	PR #7279	2107229401274912768
2026-10-05 21:04	PR #7362	2107215468136763392
2026-10-05 20:28	PR #7319	2107206250759458816
2026-10-05 19:55	PR #7365	2107198083426160640
2026-10-05 19:49	PR #7332	2107196586068021248
2026-10-05 19:39	PR #7349	2107193894448926720
2026-10-05 18:56	PR #7363	2107183062990196736
2026-10-05 17:59	PR #7342	2107168939829104640
2026-10-05 17:59	PR #7351	2107168887228338176
2026-10-05 17:59	PR #7352	2107168871638110208
2026-10-05 17:59	PR #7355	2107168836758278144
2026-10-05 17:08	PR #7325	2107155963554304000
2026-10-05 16:34	PR #7362	2107147472735309824
2026-10-05 16:21	PR #7088	2107144096983289856
2026-10-05 15:32	PR #7279	2107131932365033472
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 14:44	PR #6746	2107119792417673216
2026-10-05 14:36	PR #7359	2107117812953649152
2026-10-05 13:11	PR #7158	2107096289194807296
2026-10-05 12:56	PR #7313	2107092599713042432
2026-10-05 12:11	PR #7355	2107081303852584960
2026-10-05 11:14	PR #7351	2107066931902156800
2026-10-05 07:24	PR #7215	2107008962619510784
2026-10-05 06:58	PR #7342	2107002394956009472
2026-10-05 06:51	PR #7331	2107000790592786432
2026-10-05 04:56	PR #7345	2106971795000135680

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:42:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mchpr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:43:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mchpr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:21:13Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	29

Affected runs (29)

Date (UTC)	Associated PR	Prow job
2026-10-09 22:55	PR #7448	2108692920105177088
2026-10-09 17:34	PR #6904	2108611968167317504
2026-10-09 17:15	PR #7440	2108607415241936896
2026-10-09 11:02	PR #7437	2108513309102706688
2026-10-09 06:02	PR #7292	2108438042090934272
2026-10-09 04:09	PR #7105	2108409414854643712
2026-10-08 22:02	PR #7292	2108317188833153024
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 18:12	PR #6364	2108259306926247936
2026-10-08 18:00	PR #7389	2108256213476052992
2026-10-08 14:06	PR #7422	2108197422965460992
2026-10-08 11:12	PR #7383	2108153534082977792
2026-10-08 04:44	PR #7414	2108055963964542976
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-07 16:49	PR #7143	2107875990943305728
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 12:18	PR #7162	2107445319187828736
2026-10-06 10:37	PR #7342	2107419847431294976
2026-10-06 08:53	PR #7332	2107393708209999872
2026-10-06 02:05	PR #7356	2107291188112396288
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-05 21:45	PR #7350	2107225642733932544
2026-10-05 19:55	PR #7365	2107198083426160640
2026-10-05 19:39	PR #7349	2107193894448926720
2026-10-05 19:04	PR #7279	2107185267151474688
2026-10-05 17:59	PR #7352	2107168871638110208
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 11:11	PR #7349	2107066190932217856

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 17 time(s) (14 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-lhgsl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-lhgsl has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-tt7dl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-tt7dl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:03:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i", pod="ignition-server-proxy-7f44d8d46-qdz9k", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i/ignition-server-proxy-7f44d8d46-qdz9k has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-vp2rh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-vp2rh has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="kube-controller-manager-6fbbbcbff9-8w5vs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/kube-controller-manager-6fbbbcbff9-8w5vs has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-dfpqp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-dfpqp has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm44q5ep2osnuu42ndqjo11p7p3m6-q6t1s1d8h2q6s5k", pod="ovnkube-control-plane-c5d7b9d44-sm4sw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm44q5ep2osnuu42ndqjo11p7p3m6-q6t1s1d8h2q6s5k/ovnkube-control-plane-c5d7b9d44-sm4sw has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-5jm6z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-5jm6z has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-hrjrj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-hrjrj has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:03:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i", pod="ignition-server-proxy-7f44d8d46-8qzfv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i/ignition-server-proxy-7f44d8d46-8qzfv has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-hp7l6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-hp7l6 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-jzrt5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-jzrt5 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-lvr9j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-lvr9j has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm3l4c9fvor23h7j8oq4g14n444ug-ingress-cert-le", pod="kube-controller-manager-bf5f57fb8-bdk2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm3l4c9fvor23h7j8oq4g14n444ug-ingress-cert-le/kube-controller-manager-bf5f57fb8-bdk2x has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T00:36:32Z
  Ended:       2026-10-12T00:41:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8959360-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctccghfgvuis1e4ubo7a7ap2uim0kd-w0e2c0f4m6k0z7f", pod="cluster-network-operator-67f4c84888-hzql4", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tctccghfgvuis1e4ubo7a7ap2uim0kd-w0e2c0f4m6k0z7f/cluster-network-operator-67f4c84888-hzql4 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T21:23:26Z
  Ended:       2026-10-11T21:29:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6435200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcqfah109q9u6g2h4huthue530shv73-p6r1u7s0v8h7i2h", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcqfah109q9u6g2h4huthue530shv73-p6r1u7s0v8h7i2h/etcd-0 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	28

Affected runs (28)

Date (UTC)	Associated PR	Prow job
2026-10-11 23:29	PR #7439	2109426221556174848
2026-10-11 23:05	PR #7424	2109420238498959360
2026-10-11 19:48	PR #7457	2109370651696435200
2026-10-11 04:32	PR #6452	2109140137769177088
2026-10-10 07:19	PR #7374	2108819747389837312
2026-10-09 23:02	PR #7439	2108694594228064256
2026-10-09 14:25	PR #7248	2108564560276361216
2026-10-08 18:00	PR #7387	2108256216495951872
2026-10-08 16:13	PR #7232	2108229382379147264
2026-10-08 03:07	PR #7412	2108031501605015552
2026-10-07 20:10	PR #6364	2107926490833227776
2026-10-07 18:59	PR #7399	2107908786537631744
2026-10-07 17:18	PR #7248	2107883389313880064
2026-10-07 17:04	PR #7411	2107879778164936704
2026-10-07 14:42	PR #7410	2107844183950626816
2026-10-07 13:40	PR #6364	2107828301232869376
2026-10-07 08:33	PR #7397	2107751278686769152
2026-10-07 06:18	PR #7402	2107717187903950848
2026-10-07 03:35	PR #7400	2107676117312737280
2026-10-06 21:35	PR #7143	2107585481037320192
2026-10-06 18:48	PR #7389	2107543566250151936
2026-10-06 18:48	PR #7388	2107543517562671104
2026-10-06 16:04	PR #6191	2107502219598237696
2026-10-06 11:03	PR #7376	2107426474779742208
2026-10-06 08:09	PR #6746	2107382803778768896
2026-10-05 16:21	PR #7088	2107144096983289856
2026-10-05 11:11	PR #7292	2107066130689429504
2026-10-05 04:56	PR #7345	2106971795000135680

alert [svc] KustoLogsDataStale fired

full failure pattern:

alert [svc] KustoLogsDataStale fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:22:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:23:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire	23

Affected runs (23)

Date (UTC)	Associated PR	Prow job
2026-10-08 22:02	PR #7389	2108317197410504704
2026-10-08 22:02	PR #7364	2108317114874990592
2026-10-08 18:00	PR #7414	2108256251258343424
2026-10-08 17:29	PR #7371	2108248385097043968
2026-10-08 15:40	PR #7162	2108221120938250240
2026-10-08 13:01	PR #7387	2108181128597737472
2026-10-08 11:08	PR #6364	2108152449343688704
2026-10-08 09:41	PR #7288	2108130674367008768
2026-10-07 23:41	PR #7288	2107979694195347456
2026-10-07 17:08	PR #6807	2107880778506440704
2026-10-07 12:59	PR #7407	2107818144989974528
2026-10-07 03:35	PR #7400	2107676117312737280
2026-10-06 19:47	PR #6928	2107558478506627073
2026-10-06 14:41	PR #7385	2107481387274604544
2026-10-06 13:48	PR #7387	2107467990634074112
2026-10-06 08:53	PR #7332	2107393708209999872
2026-10-06 06:56	PR #7331	2107364414918234112
2026-10-05 21:45	PR #7350	2107225642733932544
2026-10-05 17:59	PR #7313	2107168826272518144
2026-10-05 12:34	PR #7350	2107087110770003968
2026-10-05 12:14	PR #7356	2107081906662150144
2026-10-05 11:11	PR #7349	2107066190932217856
2026-10-05 11:10	PR #7325	2107065983326752768

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-23980774-vmss000000-slpqz", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-23980774-vmss000000-slpqz has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-lzv6r", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-lzv6r has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-qbqjf", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-qbqjf has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:12:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-6f6gx", region="westus3", severity="warning"
  Description: Pod swift-recorder/swift-recorder-6f6gx has been in a non-ready state for longer than 5 minutes.

alert fired 11 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-2g4tl", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-2g4tl has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-l5mn9", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-l5mn9 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vfshb", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vfshb has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-qhgdx", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-qhgdx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-wm888", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-wm888 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-t9gmx", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-t9gmx has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-h7lkm", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-h7lkm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-w5s5h", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-w5s5h has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vwbm7", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vwbm7 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-n644b", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-n644b has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-gh9b9", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-gh9b9 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-07T23:02:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2853376-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-c4d6b786-bgkk8", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4d6b786-bgkk8 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	13

Affected runs (13)

Date (UTC)	Associated PR	Prow job
2026-10-09 14:02	PR #7397	2108558761344897024
2026-10-08 18:11	PR #7433	2108258997998981120
2026-10-07 22:28	PR #7305	2107961335642853376
2026-10-07 01:40	PR #7397	2107647281405628416
2026-10-06 12:27	PR #7383	2107447564306485248
2026-10-06 03:06	PR #6820	2107306546764451840
2026-10-06 02:05	PR #7355	2107291152553086976
2026-10-06 01:20	PR #6820	2107279728753774592
2026-10-06 00:03	PR #6820	2107260312842407936
2026-10-05 22:33	PR #6820	2107237896200130560
2026-10-05 21:05	PR #6820	2107215529679785984
2026-10-05 18:06	PR #6820	2107170660328411136
2026-10-05 01:04	PR #7178	2106913296060780544

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (3)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:13:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="cloud-network-config-controller-7847666f8b-xtgjl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="78775d9a-b0ca-4c4d-bd49-d00b20e49e68"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller-7847666f8b-xtgjl (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:13:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="csi-snapshot-controller-669f7dfb7b-68ftn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6585df9e-c94d-48b0-a688-96a3851ac4dd"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/csi-snapshot-controller-669f7dfb7b-68ftn (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:14:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="csi-snapshot-controller-669f7dfb7b-68ftn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6585df9e-c94d-48b0-a688-96a3851ac4dd"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/csi-snapshot-controller-669f7dfb7b-68ftn (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:14:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="cloud-network-config-controller-7847666f8b-xtgjl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="78775d9a-b0ca-4c4d-bd49-d00b20e49e68"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller-7847666f8b-xtgjl (controller) is in waiting state (reason: "CrashLoopBackOff").

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T07:28:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0934272-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.212:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0934272/providers/microsoft.monitor/accounts/hcps-j0934272", namespace="ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n", pod="cloud-network-config-controller-5cf98f8c95-5bd8j", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a135b966-4dab-47fb-8cc2-d75dfe7326f1"
  Description: Pod ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n/cloud-network-config-controller-5cf98f8c95-5bd8j (controller) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:01:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="cloud-network-config-controller-79b774bd6b-9pkrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6b4e41e1-2282-44a1-ab27-d2b7bffdbb7d"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller-79b774bd6b-9pkrz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:01:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="csi-snapshot-controller-7fcf9f696b-5dd5v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ade41fab-9ae3-4c1c-af6d-d25dc46782e7"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/csi-snapshot-controller-7fcf9f696b-5dd5v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="csi-snapshot-controller-7fcf9f696b-5dd5v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ade41fab-9ae3-4c1c-af6d-d25dc46782e7"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/csi-snapshot-controller-7fcf9f696b-5dd5v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="cloud-network-config-controller-79b774bd6b-9pkrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6b4e41e1-2282-44a1-ab27-d2b7bffdbb7d"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller-79b774bd6b-9pkrz (controller) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	12

Affected runs (12)

Date (UTC)	Associated PR	Prow job
2026-10-09 22:55	PR #7448	2108692920105177088
2026-10-09 06:02	PR #7292	2108438042090934272
2026-10-08 22:02	PR #7180	2108317092523544576
2026-10-08 04:44	PR #7414	2108055963964542976
2026-10-07 17:04	PR #7414	2107879778638893056
2026-10-06 18:48	PR #6191	2107543487980244992
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 11:56	PR #7332	2107439755279994880
2026-10-06 08:53	PR #7332	2107393708209999872
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 11:11	PR #7349	2107066190932217856

alert [svc] MaestroAgentMetricsProxyDown fired

full failure pattern:

alert [svc] MaestroAgentMetricsProxyDown fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:02:04Z
  Ended:       2026-10-09T15:12:09Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4897024-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T14:26:58Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5137536-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T03:29:55Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j9299456-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire	9

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-10-09 14:02	PR #7397	2108558761344897024
2026-10-09 13:37	PR #6746	2108552522435137536
2026-10-09 02:34	PR #7394	2108385613169299456
2026-10-08 14:14	PR #7414	2108199393621446656
2026-10-08 13:42	PR #7427	2108191395574452224
2026-10-07 01:40	PR #7397	2107647281405628416
2026-10-06 10:38	PR #6866	2107420154424987648
2026-10-06 07:20	PR #7332	2107370522202345472
2026-10-05 21:05	PR #6820	2107215529679785984

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x11708fbe8558>: 
    resource group "patch-tags-qgj9df44rmxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-qgj9df44rmxf" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "name": "dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "status": "Failed",
      "startTime": "2026-10-09T11:53:27.886110615Z",
      "endTime": "2026-10-09T12:17:38.071711819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a5khcdtm17dl9aq7kql2ocecgpqo still exists (deletion dispatched at 2026-10-09T11:53:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x11708fbe8558>: 
    resource group "patch-tags-qgj9df44rmxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-qgj9df44rmxf" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "name": "dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "status": "Failed",
      "startTime": "2026-10-09T11:53:27.886110615Z",
      "endTime": "2026-10-09T12:17:38.071711819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a5khcdtm17dl9aq7kql2ocecgpqo still exists (deletion dispatched at 2026-10-09T11:53:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x330cf0c08078>: 
    resource group "kusto-logs-dxkfj4jt2n4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dxkfj4jt2n4s" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "name": "735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:18.945459438Z",
      "endTime": "2026-10-09T12:19:26.457578493Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a54asvaa4eaak0rehljj280alpei still exists (deletion dispatched at 2026-10-09T11:55:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x330cf0c08078>: 
    resource group "kusto-logs-dxkfj4jt2n4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dxkfj4jt2n4s" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "name": "735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:18.945459438Z",
      "endTime": "2026-10-09T12:19:26.457578493Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a54asvaa4eaak0rehljj280alpei still exists (deletion dispatched at 2026-10-09T11:55:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2b3aa8d40060>: 
    resource group "private-keyvault-j49nnk79lk7l": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j49nnk79lk7l" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0",
      "name": "0feb53b8-520d-463a-b844-2af1832a77a0",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:12.658651084Z",
      "endTime": "2026-10-09T12:20:16.061081704Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87h9gah70vts6mbse5o06n385j3o0 still exists (deletion dispatched at 2026-10-09T11:56:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2b3aa8d40060>: 
    resource group "private-keyvault-j49nnk79lk7l": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j49nnk79lk7l" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0",
      "name": "0feb53b8-520d-463a-b844-2af1832a77a0",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:12.658651084Z",
      "endTime": "2026-10-09T12:20:16.061081704Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87h9gah70vts6mbse5o06n385j3o0 still exists (deletion dispatched at 2026-10-09T11:56:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

Contributing tests (12)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-10-09 11:00	PR #7441	2108512930990395392
2026-10-09 10:55	PR #7162	2108511709156085760
2026-10-09 02:04	PR #7414	2108377968924430336
2026-10-08 21:38	PR #6928	2108311064448864256
2026-10-08 14:53	PR #7361	2108209233152446464
2026-10-08 08:20	PR #7414	2108110245078241280
2026-10-07 16:47	PR #7400	2107875420446658560

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-2xlsbr with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2653bb4a2240>: 
    failed to create HCP cluster np-version-upgrade-cluster-2xlsbr: failed waiting for cluster="np-version-upgrade-cluster-2xlsbr" in resourcegroup="rg-np-version-upgrade-2xlsbr-hc74hngnvw7h" to finish creating: GET https://rp.j5021952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "name": "9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "status": "Failed",
      "startTime": "2026-10-09T17:24:44.640587758Z",
      "endTime": "2026-10-09T17:53:46.503674235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-2xlsbr with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2653bb4a2240>: 
    failed to create HCP cluster np-version-upgrade-cluster-2xlsbr: failed waiting for cluster="np-version-upgrade-cluster-2xlsbr" in resourcegroup="rg-np-version-upgrade-2xlsbr-hc74hngnvw7h" to finish creating: GET https://rp.j5021952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "name": "9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "status": "Failed",
      "startTime": "2026-10-09T17:24:44.640587758Z",
      "endTime": "2026-10-09T17:53:46.503674235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xa118982a360>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dx54wv2l48hl" to finish creating: GET https://rp.j4542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "name": "54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "status": "Failed",
      "startTime": "2026-10-08T05:35:12.262324182Z",
      "endTime": "2026-10-08T06:04:17.39866058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xa118982a360>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dx54wv2l48hl" to finish creating: GET https://rp.j4542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "name": "54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "status": "Failed",
      "startTime": "2026-10-08T05:35:12.262324182Z",
      "endTime": "2026-10-08T06:04:17.39866058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-wqqfmh with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x73a369a7a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-wqqfmh: failed waiting for cluster="np-version-upgrade-cluster-wqqfmh" in resourcegroup="rg-np-version-upgrade-wqqfmh-lll62bmp9wj7" to finish creating: GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40",
      "name": "4999b4cd-096d-4742-9066-ebbbba821a40",
      "status": "Failed",
      "startTime": "2026-10-06T23:10:15.924999106Z",
      "endTime": "2026-10-06T23:39:24.649289119Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-wqqfmh: failed waiting for cluster=\"np-version-upgrade-cluster-wqqfmh\" in resourcegroup=\"rg-np-version-upgrade-wqqfmh-lll62bmp9wj7\" to finish creating: GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeadlineExceeded\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"name\": \"4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-06T23:10:15.924999106Z\",\n  \"endTime\": \"2026-10-06T23:39:24.649289119Z\",\n  \"error\": {\n    \"code\": \"DeadlineExceeded\",\n    \"message\": \"cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0x73a369a780>{
            msg: "failed waiting for cluster=\"np-version-upgrade-cluster-wqqfmh\" in resourcegroup=\"rg-np-version-upgrade-wqqfmh-lll62bmp9wj7\" to finish creating: GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeadlineExceeded\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"name\": \"4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-06T23:10:15.924999106Z\",\n  \"endTime\": \"2026-10-06T23:39:24.649289119Z\",\n  \"error\": {\n    \"code\": \"DeadlineExceeded\",\n    \"message\": \"cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0x73a3797110>{
                ErrorCode: "DeadlineExceeded",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "413287f9-4023-4a70-8bd7-ff14444c1dc4",
                        ],
                        "Date": [
                            "Tue, 06 Oct 2026 23:39:31 GMT",
                        ],
                        "Content-Length": ["800"],
                        "X-Envoy-Upstream-Service-Time": ["8"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0x73a369a660>{
                        s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 101, 100, 49, 50, 50, 100, 49, 45, 55, 101, 48, 51, 45, 52, 97, 48, 49, 45, 98, 97, 97, 101, 45, 57, 48, 50, 48, 97, 98, 102, 51, 53, 48, 100, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 99, 97, 110, 97, 100, 97, 99, 101, 110, 116, 114, 97, 108, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 52, 57, 57, 57, 98, 52, 99, 100, 45, 48, 57, 54, 100, 45, 52, 55, 52, 50, 45, 57, 48, 54, 54, 45, 101, 98, 98, 98, 98, 97, 56, 50, 49, 97, 52, 48, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 52, 57, 57, 57, 98, 52, 99, 100, 45, 48, 57, 54, 100, 45, 52, 55, 52, 50, 45, 57, 48, 54, 54, 45, 101, 98, 98, 98, 98, 97, 56, 50, 49, 97, 52, 48, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 49, 48, 45, 48, 54, 84, 50, 51, 58, 49, 48, 58, 49, 53, 46, 57, 50, 52, 57, 57, 57, 49, 48, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 49, 48, 45, 48, 54, 84, 50, 51, 58, 51, 57, 58, 50, 52, 46, 54, 52, 57, 50, 56, 57, 49, 49, 57, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 68, 101, 97, 100, 108, 105, 110, 101, 69, 120, 99, 101, 101, 100, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 99, 108, 117, 115, 116, 101, 114, 32, 99, 114, 101, 97, 116, 105, 111, 110, 32, 100, 105, 100, 32, 110, 111, 116, 32, 99, 111, 109, 112, 108, 101, 116, 101, 32, 98, 101, 102, 111, 114, 101, 32, 116, 104, 101, 32, 100, 101, 97, 100, 108, 105, 110, 101, 59, 32, 91, 99, 108, 117, 115, 116, 101, 114, 83, 101, 114, 118, 105, 99, 101, 67, 108, 117, 115, 116, 101, 114, 83, 116, 97, 116, 117, 115, 93, 32, 99, 108, 117, 115, 116, 101, 114, 32, 115, 101, 114, 118, 105, 99, 101, 32, 105, 115, 32, 105, 110, 115, 116, 97, 108, 108, 105, 110, 103, 59, 32, 91, 104, 121, 112, 101, 114, 115, 104, 105, 102, 116, 72, 111, 115, 116, 101, 100, 67, 108, 117, 115, 116, 101, 114, 93, 32, 104, 111, 115, 116, 101, 100, 32, 99, 108, 117, 115, 116, 101, 114, 32, 105, 115, 32, 110, 111, 116, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 58, 32, 75, 117, 98, 101, 99, 111, 110, 102, 105, 103, 87, 97, 105, 116, 105, 110, 103, 70, 111, 114, 67, 114, 101, 97, 116, 101, 58, 32, 87, 97, 105, 116, 105, 110, 103, 32, 102, 111, 114, 32, 104, 111, 115, 116, 101, 100, 32, 99, 111, 110, 116, 114, 111, 108, 32, 112, 108, 97, 110, 101, 32, 107, 117, 98, 101, 99, 111, 110, 102, 105, 103, 32, 116, 111, 32, 98, 101, 32, 99, 114, 101, 97, 116, 101, 100, 59, 32, 104, 111, 115, 116, 101, 100, 32, 99, 108, 117, 115, 116, 101, 114, 32, 100, 101, 103, 114, 97, 100, 101, 100, 58, 32, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 82, 101, 112, 108, 105, 99, 97, 115, 58, 32, 99, 97, 112, 105, 45, 112, 114, 111, 118, 105, 100, 101, 114, 32, 100, 101, 112, 108, 111, 121, 109, 101, 110, 116, 32, 104, 97, 115, 32, 49, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 114, 101, 112, 108, 105, 99, 97, 115, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 800,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "GET",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                            Host: "rp.j6753920.hcpsvc.osadev.cloud",
                            Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40",
                            Fragment: "",
                            RawQuery: "api-version=2024-06-10-preview",
                            RawPath: "",
                            RawFragment: "",
                            ForceQuery: false,
                            OmitHost: false,
                        },
                        Proto: "HTTP/1.1",
                        ProtoMajor: 1,
                        ProtoMinor: 1,
                        Header: {
                            "X-Ms-Correlation-Request-Id": [
                                "5a323382-1250-4c09-b66e-1da47d88e137",
                            ],
                            "X-Ms-Client-Request-Id": [
                                "0dbeab77-d8c7-46ea-b31e-81602707a665",
                            ],
                            "X-Ms-Arm-Resource-System-Data": [
                                "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-10-06T23:39:31Z\"}",
                            ],
                            "X-Ms-Identity-Url": [
                                "https://dummyhost.identity.azure.net",
                            ],
                            "User-Agent": [
                                "e2e/development azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.26.7 (Red Hat 1.26.7-1.el9_8) X:strictfipsruntime; linux)",
                            ],
                            "Authorization": ["redacted"],
                        },
                        Body: nil,
                        GetBody: nil,
                        ContentLength: 0,
                        TransferEncoding: nil,
                        Close: false,
                        Host: "rp.j6753920.hcpsvc.osadev.cloud",
                        Form: nil,
                        PostForm: nil,
                        MultipartForm: nil,
                        Trailer: nil,
                        RemoteAddr: "",
                        RequestURI: "",
                        TLS: nil,
                        Cancel: nil,
                        Response: nil,
                        Pattern: "",
                        ctx: <suppressed context>,
                        pat: nil,
                        matches: nil,
                        otherValues: nil,
                    },
                    TLS: {
                        Version: 772,
                        HandshakeComplete: true,
                        DidResume: false,
                        CipherSuite: 4865,
                        CurveID: 29,
                        NegotiatedProtocol: "h2",
                        NegotiatedProtocolIsMutual: true,
                        ServerName: "rp.j6753920.hcpsvc.osadev.cloud",
                        PeerCertificates: [
                            {
                                Raw: [48, 130, 3, 136, 48, 130, 2, 112, 160, 3, 2, 1, 2, 2, 16, 101, 187, 115, 47, 59, 92, 66, 189, 190, 99, 239, 159, 123, 24, 113, 47, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 54, 50, 50, 48, 57, 48, 53, 90, 23, 13, 50, 55, 48, 52, 48, 54, 50, 50, 49, 57, 48, 53, 90, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 180, 189, 54, 60, 253, 100, 78, 1, 250, 233, 218, 220, 53, 103, 132, 235, 241, 81, 163, 211, 108, 235, 253, 159, 161, 40, 55, 174, 199, 182, 3, 193, 180, 242, 140, 167, 119, 6, 75, 110, 87, 209, 176, 132, 211, 235, 60, 11, 160, 250, 22, 226, 240, 15, 125, 1, 84, 115, 91, 242, 98, 68, 146, 207, 102, 79, 235, 209, 75, 87, 72, 253, 226, 250, 119, 155, 200, 33, 6, 113, 102, 107, 33, 217, 190, 185, 30, 162, 82, 114, 51, 161, 152, 111, 12, 189, 131, 130, 65, 170, 162, 56, 165, 235, 252, 65, 138, 22, 233, 60, 59, 235, 38, 17, 45, 43, 17, 209, 154, 75, 243, 120, 4, 159, 70, 14, 169, 120, 76, 155, 133, 47, 243, 203, 191, 154, 197, 139, 143, 159, 254, 132, 66, 214, 182, 158, 147, 234, 208, 29, 27, 123, 120, 173, 229, 243, 197, 224, 216, 119, 244, 77, 92, 73, 40, 153, 34, 241, 56, 200, 159, 205, 185, 76, 213, 2, 108, 153, 239, 202, 107, 19, 124, 230, 108, 195, 233, 21, 164, 125, 204, 173, 188, 218, 90, 132, 241, 224, 184, 206, 215, 186, 193, 201, 130, 48, 154, 253, 196, 114, 53, 239, 93, 101, 174, 230, 86, 137, 201, 61, 104, 28, 36, 165, 13, 222, 100, 63, 250, 183, 2, 129, 16, 51, 102, 254, 15, 100, 36, 182, 41, 40, 68, 69, 198, 224, 6, 92, 82, 173, 227, 191, 237, 197, 240, 61, 2, 3, 1, 0, 1, 163, 129, 169, 48, 129, 166, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 42, 6, 3, 85, 29, 17, 4, 35, 48, 33, 130, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 159, 109, 142, 74, 144, 119, 103, 164, 31, 21, 250, 192, 45, 187, 81, 133, 130, 67, 122, 179, 175, 201, 28, 155, 113, 111, 195, 186, 204, 11, 27, 164, 12, 102, 154, 125, 210, 179, 147, 165, 234, 178, 47, 253, 221, 134, 200, 195, 217, 37, 230, 239, 211, 172, 74, 210, 218, 4, 225, 62, 131, 100, 100, 85, 196, 87, 233, 126, 51, 132, 162, 119, 216, 5, 31, 227, 121, 41, 23, 46, 0, 239, 160, 98, 149, 33, 175, 64, 235, 157, 70, 131, 247, 19, 178, 243, 154, 96, 146, 109, 89, 156, 215, 163, 11, 15, 49, 24, 36, 77, 124, 8, 138, 96, 135, 55, 31, 78, 93, 70, 187, 170, 173, 175, 52, 74, 140, 92, 128, 33, 119, 209, 243, 175, 98, 152, 220, 11, 129, 142, 148, 160, 192, 255, 19, 24, 220, 176, 117, 9, 123, 202, 239, 60, 180, 86, 191, 240, 240, 59, 218, 219, 249, 189, 10, 211, 181, 171, 64, 11, 156, 121, 109, 73, 216, 246, 199, 210, 25, 70, 57, 145, 122, 92, 184, 246, 255, 83, 160, 81, 243, 186, 178, 9, 91, 46, 150, 189, 173, 253, 79, 153, 208, 236, 144, 241, 13, 30, 175, 210, 56, 128, 140, 40, 150, 32, 46, 81, 121, 233, 100, 117, 223, 100, 218, 1, 11, 104, 77, 177, 162, 73, 109, 32, 185, 154, 57, 86, 114, 201, 230, 183, 115, 247, 149, 5, 162, 201, 105, 66, 31, 183, 62, 66, 163, 154],
                                RawTBSCertificate: [48, 130, 2, 112, 160, 3, 2, 1, 2, 2, 16, 101, 187, 115, 47, 59, 92, 66, 189, 190, 99, 239, 159, 123, 24, 113, 47, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 54, 50, 50, 48, 57, 48, 53, 90, 23, 13, 50, 55, 48, 52, 48, 54, 50, 50, 49, 57, 48, 53, 90, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 180, 189, 54, 60, 253, 100, 78, 1, 250, 233, 218, 220, 53, 103, 132, 235, 241, 81, 163, 211, 108, 235, 253, 159, 161, 40, 55, 174, 199, 182, 3, 193, 180, 242, 140, 167, 119, 6, 75, 110, 87, 209, 176, 132, 211, 235, 60, 11, 160, 250, 22, 226, 240, 15, 125, 1, 84, 115, 91, 242, 98, 68, 146, 207, 102, 79, 235, 209, 75, 87, 72, 253, 226, 250, 119, 155, 200, 33, 6, 113, 102, 107, 33, 217, 190, 185, 30, 162, 82, 114, 51, 161, 152, 111, 12, 189, 131, 130, 65, 170, 162, 56, 165, 235, 252, 65, 138, 22, 233, 60, 59, 235, 38, 17, 45, 43, 17, 209, 154, 75, 243, 120, 4, 159, 70, 14, 169, 120, 76, 155, 133, 47, 243, 203, 191, 154, 197, 139, 143, 159, 254, 132, 66, 214, 182, 158, 147, 234, 208, 29, 27, 123, 120, 173, 229, 243, 197, 224, 216, 119, 244, 77, 92, 73, 40, 153, 34, 241, 56, 200, 159, 205, 185, 76, 213, 2, 108, 153, 239, 202, 107, 19, 124, 230, 108, 195, 233, 21, 164, 125, 204, 173, 188, 218, 90, 132, 241, 224, 184, 206, 215, 186, 193, 201, 130, 48, 154, 253, 196, 114, 53, 239, 93, 101, 174, 230, 86, 137, 201, 61, 104, 28, 36, 165, 13, 222, 100, 63, 250, 183, 2, 129, 16, 51, 102, 254, 15, 100, 36, 182, 41, 40, 68, 69, 198, 224, 6, 92, 82, 173, 227, 191, 237, 197, 240, 61, 2, 3, 1, 0, 1, 163, 129, 169, 48, 129, 166, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 42, 6, 3, 85, 29, 17, 4, 35, 48, 33, 130, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154],
                                RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 180, 189, 54, 60, 253, 100, 78, 1, 250, 233, 218, 220, 53, 103, 132, 235, 241, 81, 163, 211, 108, 235, 253, 159, 161, 40, 55, 174, 199, 182, 3, 193, 180, 242, 140, 167, 119, 6, 75, 110, 87, 209, 176, 132, 211, 235, 60, 11, 160, 250, 22, 226, 240, 15, 125, 1, 84, 115, 91, 242, 98, 68, 146, 207, 102, 79, 235, 209, 75, 87, 72, 253, 226, 250, 119, 155, 200, 33, 6, 113, 102, 107, 33, 217, 190, 185, 30, 162, 82, 114, 51, 161, 152, 111, 12, 189, 131, 130, 65, 170, 162, 56, 165, 235, 252, 65, 138, 22, 233, 60, 59, 235, 38, 17, 45, 43, 17, 209, 154, 75, 243, 120, 4, 159, 70, 14, 169, 120, 76, 155, 133, 47, 243, 203, 191, 154, 197, 139, 143, 159, 254, 132, 66, 214, 182, 158, 147, 234, 208, 29, 27, 123, 120, 173, 229, 243, 197, 224, 216, 119, 244, 77, 92, 73, 40, 153, 34, 241, 56, 200, 159, 205, 185, 76, 213, 2, 108, 153, 239, 202, 107, 19, 124, 230, 108, 195, 233, 21, 164, 125, 204, 173, 188, 218, 90, 132, 241, 224, 184, 206, 215, 186, 193, 201, 130, 48, 154, 253, 196, 114, 53, 239, 93, 101, 174, 230, 86, 137, 201, 61, 104, 28, 36, 165, 13, 222, 100, 63, 250, 183, 2, 129, 16, 51, 102, 254, 15, 100, 36, 182, 41, 40, 68, 69, 198, 224, 6, 92, 82, 173, 227, 191, 237, 197, 240, 61, 2, 3, 1, 0, 1],
                                RawSubject: [48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                                RawIssuer: [48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                                Signature: [159, 109, 142, 74, 144, 119, 103, 164, 31, 21, 250, 192, 45, 187, 81, 133, 130, 67, 122, 179, 175, 201, 28, 155, 113, 111, 195, 186, 204, 11, 27, 164, 12, 102, 154, 125, 210, 179, 147, 165, 234, 178, 47, 253, 221, 134, 200, 195, 217, 37, 230, 239, 211, 172, 74, 210, 218, 4, 225, 62, 131, 100, 100, 85, 196, 87, 233, 126, 51, 132, 162, 119, 216, 5, 31, 227, 121, 41, 23, 46, 0, 239, 160, 98, 149, 33, 175, 64, 235, 157, 70, 131, 247, 19, 178, 243, 154, 96, 146, 109, 89, 156, 215, 163, 11, 15, 49, 24, 36, 77, 124, 8, 138, 96, 135, 55, 31, 78, 93, 70, 187, 170, 173, 175, 52, 74, 140, 92, 128, 33, 119, 209, 243, 175, 98, 152, 220, 11, 129, 142, 148, 160, 192, 255, 19, 24, 220, 176, 117, 9, 123, 202, 239, 60, 180, 86, 191, 240, 240, 59, 218, 219, 249, 189, 10, 211, 181, 171, 64, 11, 156, 121, 109, 73, 216, 246, 199, 210, 25, 70, 57, 145, 122, 92, 184, 246, 255, 83, 160, 81, 243, 186, 178, 9, 91, 46, 150, 189, 173, 253, 79, 153, 208, 236, 144, 241, 13, 30, 175, 210, 56, 128, 140, 40, 150, 32, 46, 81, 121, 233, 100, 117, 223, 100, 218, 1, 11, 104, 77, 177, 162, 73, 109, 32, 185, 154, 57, 86, 114, 201, 230, 183, 115, 247, 149, 5, 162, 201, 105, 66, 31, 183, 62, 66, 163, 154],
                                SignatureAlgorithm: 4,
                                PublicKeyAlgorithm: 1,
                                PublicKey: <*rsa.PublicKey | 0x73a3286f20>{
                                    N: {
                                        neg: false,
                                        abs: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    E: 65537,
                                },
                                Version: 3,
                                SerialNumber: {
                                    neg: false,
                                    abs: [13719072358144700719, 7330579465172173501],
                                },
                                Issuer: {
                                    Country: nil,
                                    Organization: nil,
                                    OrganizationalUnit: nil,
                                    Locality: nil,
                                    Province: nil,
                                    StreetAddress: nil,
                                    PostalCode: nil,
                                    SerialNumber: "",
                                    CommonName: "rp.j6753920.hcpsvc.osadev.cloud",
                                    Names: [
                                        {Type: ..., Value: ...},
                                    ],
                                    ExtraNames: nil,
                                },
                                Subject: {
                                    Country: nil,
                                    Organization: nil,
                                    OrganizationalUnit: nil,
                                    Locality: nil,
                                    Province: nil,
                                    StreetAddress: nil,
                                    PostalCode: nil,
                                    SerialNumber: "",
                                    CommonName: "rp.j6753920.hcpsvc.osadev.cloud",
                                    Names: [
                                        {Type: ..., Value: ...},
                                    ],
                                    ExtraNames: nil,
                                },
                                NotBefore: {wall: 0, ext: 63926921345, loc: nil},
                                NotAfter: {wall: 0, ext: 63942646745, loc: nil},
                                KeyUsage: 5,
                                Extensions: [
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: true,
                                        Value: [..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                ],
                                ExtraExtensions: nil,
                                UnhandledCriticalExtensions: nil,
                                ExtKeyUsage: [1, 2],
                                UnknownExtKeyUsage: nil,
                                BasicConstraintsValid: true,
                                IsCA: false,
                                MaxPathLen: -1,
                                MaxPathLenZero: false,
                                SubjectKeyId: [125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154],
                                AuthorityKeyId: [125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154],
                                OCSPServer: nil,
                                IssuingCertificateURL: nil,
                                DNSNames: [
                                    "rp.j6753920.hcpsvc.osadev.cloud",
                                ],
                                EmailAddresses: nil,
                                IPAddresses: nil,
                                URIs: nil,
                                PermittedDNSDomainsCritical: false,
                                PermittedDNSDomains: nil,
                                ExcludedDNSDomains: nil,
                                PermittedIPRanges: nil,
                                ExcludedIPRanges: nil,
                                PermittedEmailAddresses: nil,
                                ExcludedEmailAddresses: nil,
                                PermittedURIDomains: nil,
                                ExcludedURIDomains: nil,
                                CRLDistributionPoints: nil,
                                PolicyIdentifiers: nil,
                                Policies: nil,
                                InhibitAnyPolicy: 0,
                                InhibitAnyPolicyZero: false,
                                InhibitPolicyMapping: 0,
                                InhibitPolicyMappingZero: false,
                                RequireExplicitPolicy: 0,
                                RequireExplicitPolicyZero: false,
                                PolicyMappings: nil,
                            },
                        ],
                        VerifiedChains: nil,
                        SignedCertificateTimestamps: nil,
                        OCSPResponse: nil,
                        TLSUnique: nil,
                        ECHAccepted: false,
                        HelloRetryRequest: false,
                        ekm: 0x585e900,
                        testingOnlyPeerSignatureAlgorithm: 2052,
                    },
                },
                errMsg: "GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeadlineExceeded\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"name\": \"4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-06T23:10:15.924999106Z\",\n  \"endTime\": \"2026-10-06T23:39:24.649289119Z\",\n  \"error\": {\n    \"code\": \"DeadlineExceeded\",\n    \"message\": \"cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-wqqfmh with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x73a369a7a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-wqqfmh: failed waiting for cluster="np-version-upgrade-cluster-wqqfmh" in resourcegroup="rg-np-version-upgrade-wqqfmh-lll62bmp9wj7" to finish creating: GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40",
      "name": "4999b4cd-096d-4742-9066-ebbbba821a40",
      "status": "Failed",
      "startTime": "2026-10-06T23:10:15.924999106Z",
      "endTime": "2026-10-06T23:39:24.649289119Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-wqqfmh: failed waiting for cluster=\"np-version-upgrade-cluster-wqqfmh\" in resourcegroup=\"rg-np-version-upgrade-wqqfmh-lll62bmp9wj7\" to finish creating: GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeadlineExceeded\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"name\": \"4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-06T23:10:15.924999106Z\",\n  \"endTime\": \"2026-10-06T23:39:24.649289119Z\",\n  \"error\": {\n    \"code\": \"DeadlineExceeded\",\n    \"message\": \"cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0x73a369a780>{
            msg: "failed waiting for cluster=\"np-version-upgrade-cluster-wqqfmh\" in resourcegroup=\"rg-np-version-upgrade-wqqfmh-lll62bmp9wj7\" to finish creating: GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeadlineExceeded\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"name\": \"4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-06T23:10:15.924999106Z\",\n  \"endTime\": \"2026-10-06T23:39:24.649289119Z\",\n  \"error\": {\n    \"code\": \"DeadlineExceeded\",\n    \"message\": \"cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0x73a3797110>{
                ErrorCode: "DeadlineExceeded",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "413287f9-4023-4a70-8bd7-ff14444c1dc4",
                        ],
                        "Date": [
                            "Tue, 06 Oct 2026 23:39:31 GMT",
                        ],
                        "Content-Length": ["800"],
                        "X-Envoy-Upstream-Service-Time": ["8"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0x73a369a660>{
                        s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 101, 100, 49, 50, 50, 100, 49, 45, 55, 101, 48, 51, 45, 52, 97, 48, 49, 45, 98, 97, 97, 101, 45, 57, 48, 50, 48, 97, 98, 102, 51, 53, 48, 100, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 99, 97, 110, 97, 100, 97, 99, 101, 110, 116, 114, 97, 108, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 52, 57, 57, 57, 98, 52, 99, 100, 45, 48, 57, 54, 100, 45, 52, 55, 52, 50, 45, 57, 48, 54, 54, 45, 101, 98, 98, 98, 98, 97, 56, 50, 49, 97, 52, 48, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 52, 57, 57, 57, 98, 52, 99, 100, 45, 48, 57, 54, 100, 45, 52, 55, 52, 50, 45, 57, 48, 54, 54, 45, 101, 98, 98, 98, 98, 97, 56, 50, 49, 97, 52, 48, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 49, 48, 45, 48, 54, 84, 50, 51, 58, 49, 48, 58, 49, 53, 46, 57, 50, 52, 57, 57, 57, 49, 48, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 49, 48, 45, 48, 54, 84, 50, 51, 58, 51, 57, 58, 50, 52, 46, 54, 52, 57, 50, 56, 57, 49, 49, 57, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 68, 101, 97, 100, 108, 105, 110, 101, 69, 120, 99, 101, 101, 100, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 99, 108, 117, 115, 116, 101, 114, 32, 99, 114, 101, 97, 116, 105, 111, 110, 32, 100, 105, 100, 32, 110, 111, 116, 32, 99, 111, 109, 112, 108, 101, 116, 101, 32, 98, 101, 102, 111, 114, 101, 32, 116, 104, 101, 32, 100, 101, 97, 100, 108, 105, 110, 101, 59, 32, 91, 99, 108, 117, 115, 116, 101, 114, 83, 101, 114, 118, 105, 99, 101, 67, 108, 117, 115, 116, 101, 114, 83, 116, 97, 116, 117, 115, 93, 32, 99, 108, 117, 115, 116, 101, 114, 32, 115, 101, 114, 118, 105, 99, 101, 32, 105, 115, 32, 105, 110, 115, 116, 97, 108, 108, 105, 110, 103, 59, 32, 91, 104, 121, 112, 101, 114, 115, 104, 105, 102, 116, 72, 111, 115, 116, 101, 100, 67, 108, 117, 115, 116, 101, 114, 93, 32, 104, 111, 115, 116, 101, 100, 32, 99, 108, 117, 115, 116, 101, 114, 32, 105, 115, 32, 110, 111, 116, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 58, 32, 75, 117, 98, 101, 99, 111, 110, 102, 105, 103, 87, 97, 105, 116, 105, 110, 103, 70, 111, 114, 67, 114, 101, 97, 116, 101, 58, 32, 87, 97, 105, 116, 105, 110, 103, 32, 102, 111, 114, 32, 104, 111, 115, 116, 101, 100, 32, 99, 111, 110, 116, 114, 111, 108, 32, 112, 108, 97, 110, 101, 32, 107, 117, 98, 101, 99, 111, 110, 102, 105, 103, 32, 116, 111, 32, 98, 101, 32, 99, 114, 101, 97, 116, 101, 100, 59, 32, 104, 111, 115, 116, 101, 100, 32, 99, 108, 117, 115, 116, 101, 114, 32, 100, 101, 103, 114, 97, 100, 101, 100, 58, 32, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 82, 101, 112, 108, 105, 99, 97, 115, 58, 32, 99, 97, 112, 105, 45, 112, 114, 111, 118, 105, 100, 101, 114, 32, 100, 101, 112, 108, 111, 121, 109, 101, 110, 116, 32, 104, 97, 115, 32, 49, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 114, 101, 112, 108, 105, 99, 97, 115, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                        i: 0,
                    },
                    ContentLength: 800,
                    TransferEncoding: nil,
                    Close: false,
                    Uncompressed: false,
                    Trailer: nil,
                    Request: {
                        Method: "GET",
                        URL: {
                            Scheme: "https",
                            Opaque: "",
                            User: nil,
                            Host: "rp.j6753920.hcpsvc.osadev.cloud",
                            Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40",
                            Fragment: "",
                            RawQuery: "api-version=2024-06-10-preview",
                            RawPath: "",
                            RawFragment: "",
                            ForceQuery: false,
                            OmitHost: false,
                        },
                        Proto: "HTTP/1.1",
                        ProtoMajor: 1,
                        ProtoMinor: 1,
                        Header: {
                            "X-Ms-Correlation-Request-Id": [
                                "5a323382-1250-4c09-b66e-1da47d88e137",
                            ],
                            "X-Ms-Client-Request-Id": [
                                "0dbeab77-d8c7-46ea-b31e-81602707a665",
                            ],
                            "X-Ms-Arm-Resource-System-Data": [
                                "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-10-06T23:39:31Z\"}",
                            ],
                            "X-Ms-Identity-Url": [
                                "https://dummyhost.identity.azure.net",
                            ],
                            "User-Agent": [
                                "e2e/development azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.26.7 (Red Hat 1.26.7-1.el9_8) X:strictfipsruntime; linux)",
                            ],
                            "Authorization": ["redacted"],
                        },
                        Body: nil,
                        GetBody: nil,
                        ContentLength: 0,
                        TransferEncoding: nil,
                        Close: false,
                        Host: "rp.j6753920.hcpsvc.osadev.cloud",
                        Form: nil,
                        PostForm: nil,
                        MultipartForm: nil,
                        Trailer: nil,
                        RemoteAddr: "",
                        RequestURI: "",
                        TLS: nil,
                        Cancel: nil,
                        Response: nil,
                        Pattern: "",
                        ctx: <suppressed context>,
                        pat: nil,
                        matches: nil,
                        otherValues: nil,
                    },
                    TLS: {
                        Version: 772,
                        HandshakeComplete: true,
                        DidResume: false,
                        CipherSuite: 4865,
                        CurveID: 29,
                        NegotiatedProtocol: "h2",
                        NegotiatedProtocolIsMutual: true,
                        ServerName: "rp.j6753920.hcpsvc.osadev.cloud",
                        PeerCertificates: [
                            {
                                Raw: [48, 130, 3, 136, 48, 130, 2, 112, 160, 3, 2, 1, 2, 2, 16, 101, 187, 115, 47, 59, 92, 66, 189, 190, 99, 239, 159, 123, 24, 113, 47, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 54, 50, 50, 48, 57, 48, 53, 90, 23, 13, 50, 55, 48, 52, 48, 54, 50, 50, 49, 57, 48, 53, 90, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 180, 189, 54, 60, 253, 100, 78, 1, 250, 233, 218, 220, 53, 103, 132, 235, 241, 81, 163, 211, 108, 235, 253, 159, 161, 40, 55, 174, 199, 182, 3, 193, 180, 242, 140, 167, 119, 6, 75, 110, 87, 209, 176, 132, 211, 235, 60, 11, 160, 250, 22, 226, 240, 15, 125, 1, 84, 115, 91, 242, 98, 68, 146, 207, 102, 79, 235, 209, 75, 87, 72, 253, 226, 250, 119, 155, 200, 33, 6, 113, 102, 107, 33, 217, 190, 185, 30, 162, 82, 114, 51, 161, 152, 111, 12, 189, 131, 130, 65, 170, 162, 56, 165, 235, 252, 65, 138, 22, 233, 60, 59, 235, 38, 17, 45, 43, 17, 209, 154, 75, 243, 120, 4, 159, 70, 14, 169, 120, 76, 155, 133, 47, 243, 203, 191, 154, 197, 139, 143, 159, 254, 132, 66, 214, 182, 158, 147, 234, 208, 29, 27, 123, 120, 173, 229, 243, 197, 224, 216, 119, 244, 77, 92, 73, 40, 153, 34, 241, 56, 200, 159, 205, 185, 76, 213, 2, 108, 153, 239, 202, 107, 19, 124, 230, 108, 195, 233, 21, 164, 125, 204, 173, 188, 218, 90, 132, 241, 224, 184, 206, 215, 186, 193, 201, 130, 48, 154, 253, 196, 114, 53, 239, 93, 101, 174, 230, 86, 137, 201, 61, 104, 28, 36, 165, 13, 222, 100, 63, 250, 183, 2, 129, 16, 51, 102, 254, 15, 100, 36, 182, 41, 40, 68, 69, 198, 224, 6, 92, 82, 173, 227, 191, 237, 197, 240, 61, 2, 3, 1, 0, 1, 163, 129, 169, 48, 129, 166, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 42, 6, 3, 85, 29, 17, 4, 35, 48, 33, 130, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 159, 109, 142, 74, 144, 119, 103, 164, 31, 21, 250, 192, 45, 187, 81, 133, 130, 67, 122, 179, 175, 201, 28, 155, 113, 111, 195, 186, 204, 11, 27, 164, 12, 102, 154, 125, 210, 179, 147, 165, 234, 178, 47, 253, 221, 134, 200, 195, 217, 37, 230, 239, 211, 172, 74, 210, 218, 4, 225, 62, 131, 100, 100, 85, 196, 87, 233, 126, 51, 132, 162, 119, 216, 5, 31, 227, 121, 41, 23, 46, 0, 239, 160, 98, 149, 33, 175, 64, 235, 157, 70, 131, 247, 19, 178, 243, 154, 96, 146, 109, 89, 156, 215, 163, 11, 15, 49, 24, 36, 77, 124, 8, 138, 96, 135, 55, 31, 78, 93, 70, 187, 170, 173, 175, 52, 74, 140, 92, 128, 33, 119, 209, 243, 175, 98, 152, 220, 11, 129, 142, 148, 160, 192, 255, 19, 24, 220, 176, 117, 9, 123, 202, 239, 60, 180, 86, 191, 240, 240, 59, 218, 219, 249, 189, 10, 211, 181, 171, 64, 11, 156, 121, 109, 73, 216, 246, 199, 210, 25, 70, 57, 145, 122, 92, 184, 246, 255, 83, 160, 81, 243, 186, 178, 9, 91, 46, 150, 189, 173, 253, 79, 153, 208, 236, 144, 241, 13, 30, 175, 210, 56, 128, 140, 40, 150, 32, 46, 81, 121, 233, 100, 117, 223, 100, 218, 1, 11, 104, 77, 177, 162, 73, 109, 32, 185, 154, 57, 86, 114, 201, 230, 183, 115, 247, 149, 5, 162, 201, 105, 66, 31, 183, 62, 66, 163, 154],
                                RawTBSCertificate: [48, 130, 2, 112, 160, 3, 2, 1, 2, 2, 16, 101, 187, 115, 47, 59, 92, 66, 189, 190, 99, 239, 159, 123, 24, 113, 47, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 49, 48, 48, 54, 50, 50, 48, 57, 48, 53, 90, 23, 13, 50, 55, 48, 52, 48, 54, 50, 50, 49, 57, 48, 53, 90, 48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 180, 189, 54, 60, 253, 100, 78, 1, 250, 233, 218, 220, 53, 103, 132, 235, 241, 81, 163, 211, 108, 235, 253, 159, 161, 40, 55, 174, 199, 182, 3, 193, 180, 242, 140, 167, 119, 6, 75, 110, 87, 209, 176, 132, 211, 235, 60, 11, 160, 250, 22, 226, 240, 15, 125, 1, 84, 115, 91, 242, 98, 68, 146, 207, 102, 79, 235, 209, 75, 87, 72, 253, 226, 250, 119, 155, 200, 33, 6, 113, 102, 107, 33, 217, 190, 185, 30, 162, 82, 114, 51, 161, 152, 111, 12, 189, 131, 130, 65, 170, 162, 56, 165, 235, 252, 65, 138, 22, 233, 60, 59, 235, 38, 17, 45, 43, 17, 209, 154, 75, 243, 120, 4, 159, 70, 14, 169, 120, 76, 155, 133, 47, 243, 203, 191, 154, 197, 139, 143, 159, 254, 132, 66, 214, 182, 158, 147, 234, 208, 29, 27, 123, 120, 173, 229, 243, 197, 224, 216, 119, 244, 77, 92, 73, 40, 153, 34, 241, 56, 200, 159, 205, 185, 76, 213, 2, 108, 153, 239, 202, 107, 19, 124, 230, 108, 195, 233, 21, 164, 125, 204, 173, 188, 218, 90, 132, 241, 224, 184, 206, 215, 186, 193, 201, 130, 48, 154, 253, 196, 114, 53, 239, 93, 101, 174, 230, 86, 137, 201, 61, 104, 28, 36, 165, 13, 222, 100, 63, 250, 183, 2, 129, 16, 51, 102, 254, 15, 100, 36, 182, 41, 40, 68, 69, 198, 224, 6, 92, 82, 173, 227, 191, 237, 197, 240, 61, 2, 3, 1, 0, 1, 163, 129, 169, 48, 129, 166, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 42, 6, 3, 85, 29, 17, 4, 35, 48, 33, 130, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154],
                                RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 180, 189, 54, 60, 253, 100, 78, 1, 250, 233, 218, 220, 53, 103, 132, 235, 241, 81, 163, 211, 108, 235, 253, 159, 161, 40, 55, 174, 199, 182, 3, 193, 180, 242, 140, 167, 119, 6, 75, 110, 87, 209, 176, 132, 211, 235, 60, 11, 160, 250, 22, 226, 240, 15, 125, 1, 84, 115, 91, 242, 98, 68, 146, 207, 102, 79, 235, 209, 75, 87, 72, 253, 226, 250, 119, 155, 200, 33, 6, 113, 102, 107, 33, 217, 190, 185, 30, 162, 82, 114, 51, 161, 152, 111, 12, 189, 131, 130, 65, 170, 162, 56, 165, 235, 252, 65, 138, 22, 233, 60, 59, 235, 38, 17, 45, 43, 17, 209, 154, 75, 243, 120, 4, 159, 70, 14, 169, 120, 76, 155, 133, 47, 243, 203, 191, 154, 197, 139, 143, 159, 254, 132, 66, 214, 182, 158, 147, 234, 208, 29, 27, 123, 120, 173, 229, 243, 197, 224, 216, 119, 244, 77, 92, 73, 40, 153, 34, 241, 56, 200, 159, 205, 185, 76, 213, 2, 108, 153, 239, 202, 107, 19, 124, 230, 108, 195, 233, 21, 164, 125, 204, 173, 188, 218, 90, 132, 241, 224, 184, 206, 215, 186, 193, 201, 130, 48, 154, 253, 196, 114, 53, 239, 93, 101, 174, 230, 86, 137, 201, 61, 104, 28, 36, 165, 13, 222, 100, 63, 250, 183, 2, 129, 16, 51, 102, 254, 15, 100, 36, 182, 41, 40, 68, 69, 198, 224, 6, 92, 82, 173, 227, 191, 237, 197, 240, 61, 2, 3, 1, 0, 1],
                                RawSubject: [48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                                RawIssuer: [48, 42, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 114, 112, 46, 106, 54, 55, 53, 51, 57, 50, 48, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100],
                                Signature: [159, 109, 142, 74, 144, 119, 103, 164, 31, 21, 250, 192, 45, 187, 81, 133, 130, 67, 122, 179, 175, 201, 28, 155, 113, 111, 195, 186, 204, 11, 27, 164, 12, 102, 154, 125, 210, 179, 147, 165, 234, 178, 47, 253, 221, 134, 200, 195, 217, 37, 230, 239, 211, 172, 74, 210, 218, 4, 225, 62, 131, 100, 100, 85, 196, 87, 233, 126, 51, 132, 162, 119, 216, 5, 31, 227, 121, 41, 23, 46, 0, 239, 160, 98, 149, 33, 175, 64, 235, 157, 70, 131, 247, 19, 178, 243, 154, 96, 146, 109, 89, 156, 215, 163, 11, 15, 49, 24, 36, 77, 124, 8, 138, 96, 135, 55, 31, 78, 93, 70, 187, 170, 173, 175, 52, 74, 140, 92, 128, 33, 119, 209, 243, 175, 98, 152, 220, 11, 129, 142, 148, 160, 192, 255, 19, 24, 220, 176, 117, 9, 123, 202, 239, 60, 180, 86, 191, 240, 240, 59, 218, 219, 249, 189, 10, 211, 181, 171, 64, 11, 156, 121, 109, 73, 216, 246, 199, 210, 25, 70, 57, 145, 122, 92, 184, 246, 255, 83, 160, 81, 243, 186, 178, 9, 91, 46, 150, 189, 173, 253, 79, 153, 208, 236, 144, 241, 13, 30, 175, 210, 56, 128, 140, 40, 150, 32, 46, 81, 121, 233, 100, 117, 223, 100, 218, 1, 11, 104, 77, 177, 162, 73, 109, 32, 185, 154, 57, 86, 114, 201, 230, 183, 115, 247, 149, 5, 162, 201, 105, 66, 31, 183, 62, 66, 163, 154],
                                SignatureAlgorithm: 4,
                                PublicKeyAlgorithm: 1,
                                PublicKey: <*rsa.PublicKey | 0x73a3286f20>{
                                    N: {
                                        neg: false,
                                        abs: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    E: 65537,
                                },
                                Version: 3,
                                SerialNumber: {
                                    neg: false,
                                    abs: [13719072358144700719, 7330579465172173501],
                                },
                                Issuer: {
                                    Country: nil,
                                    Organization: nil,
                                    OrganizationalUnit: nil,
                                    Locality: nil,
                                    Province: nil,
                                    StreetAddress: nil,
                                    PostalCode: nil,
                                    SerialNumber: "",
                                    CommonName: "rp.j6753920.hcpsvc.osadev.cloud",
                                    Names: [
                                        {Type: ..., Value: ...},
                                    ],
                                    ExtraNames: nil,
                                },
                                Subject: {
                                    Country: nil,
                                    Organization: nil,
                                    OrganizationalUnit: nil,
                                    Locality: nil,
                                    Province: nil,
                                    StreetAddress: nil,
                                    PostalCode: nil,
                                    SerialNumber: "",
                                    CommonName: "rp.j6753920.hcpsvc.osadev.cloud",
                                    Names: [
                                        {Type: ..., Value: ...},
                                    ],
                                    ExtraNames: nil,
                                },
                                NotBefore: {wall: 0, ext: 63926921345, loc: nil},
                                NotAfter: {wall: 0, ext: 63942646745, loc: nil},
                                KeyUsage: 5,
                                Extensions: [
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: true,
                                        Value: [..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                    {
                                        Id: [..., ..., ..., ...],
                                        Critical: false,
                                        Value: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                    },
                                ],
                                ExtraExtensions: nil,
                                UnhandledCriticalExtensions: nil,
                                ExtKeyUsage: [1, 2],
                                UnknownExtKeyUsage: nil,
                                BasicConstraintsValid: true,
                                IsCA: false,
                                MaxPathLen: -1,
                                MaxPathLenZero: false,
                                SubjectKeyId: [125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154],
                                AuthorityKeyId: [125, 204, 253, 223, 155, 246, 165, 12, 115, 178, 131, 57, 252, 194, 236, 91, 243, 51, 106, 154],
                                OCSPServer: nil,
                                IssuingCertificateURL: nil,
                                DNSNames: [
                                    "rp.j6753920.hcpsvc.osadev.cloud",
                                ],
                                EmailAddresses: nil,
                                IPAddresses: nil,
                                URIs: nil,
                                PermittedDNSDomainsCritical: false,
                                PermittedDNSDomains: nil,
                                ExcludedDNSDomains: nil,
                                PermittedIPRanges: nil,
                                ExcludedIPRanges: nil,
                                PermittedEmailAddresses: nil,
                                ExcludedEmailAddresses: nil,
                                PermittedURIDomains: nil,
                                ExcludedURIDomains: nil,
                                CRLDistributionPoints: nil,
                                PolicyIdentifiers: nil,
                                Policies: nil,
                                InhibitAnyPolicy: 0,
                                InhibitAnyPolicyZero: false,
                                InhibitPolicyMapping: 0,
                                InhibitPolicyMappingZero: false,
                                RequireExplicitPolicy: 0,
                                RequireExplicitPolicyZero: false,
                                PolicyMappings: nil,
                            },
                        ],
                        VerifiedChains: nil,
                        SignedCertificateTimestamps: nil,
                        OCSPResponse: nil,
                        TLSUnique: nil,
                        ECHAccepted: false,
                        HelloRetryRequest: false,
                        ekm: 0x585e900,
                        testingOnlyPeerSignatureAlgorithm: 2052,
                    },
                },
                errMsg: "GET https://rp.j6753920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeadlineExceeded\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"name\": \"4999b4cd-096d-4742-9066-ebbbba821a40\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-06T23:10:15.924999106Z\",\n  \"endTime\": \"2026-10-06T23:39:24.649289119Z\",\n  \"error\": {\n    \"code\": \"DeadlineExceeded\",\n    \"message\": \"cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        },
    }
occurred

Contributing tests (6)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	1

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-10-09 16:02	PR #7227	2108589017825021952
2026-10-08 04:44	PR #7414	2108055963964542976
2026-10-06 21:46	PR #7389	2107588315996753920
2026-10-06 14:39	PR #7388	2107480778416852992
2026-10-06 01:06	PR #6746	2107276252678197248
2026-10-05 15:08	PR #7331	2107125784404561920
2026-10-05 11:11	PR #7349	2107066190932217856