<
Single Day: Jul 22
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
tls: failed to verify certificate: x509: certificate signed by unknown authority
23:00:46 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:186]: failed to verify simple web app runs on cluster "cluster-nightly-4-23-dpbrz9"
Unexpected error:
    <*fmt.wrapError | 0xc000fbe3e0>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-zszzt.apps.aro.x0o8h6x6l7k9u1f.e0w5.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:186]: failed to verify simple web app runs on cluster "cluster-nightly-4-23-dpbrz9"
Unexpected error:
    <*fmt.wrapError | 0xc000fbe3e0>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-zszzt.apps.aro.x0o8h6x6l7k9u1f.e0w5.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
cluster entered terminal Failed state after role assignment deployment
21:11:01 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
21:04:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-qcvwjv" for cluster "cluster-candidate-4-20-qcvwjv"
Unexpected error:
    <*fmt.wrapError | 0xc0026a6940>: 
    failed to create NodePool np-qcvwjv: failed waiting for nodepool="np-qcvwjv" for cluster "cluster-candidate-4-20-qcvwjv" in resourcegroup="rg-candidate-4-20-kwkc79fpjrx2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/a9e953c4-caff-45c7-a41d-8606a89637ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/a9e953c4-caff-45c7-a41d-8606a89637ea",
      "name": "a9e953c4-caff-45c7-a41d-8606a89637ea",
      "status": "Failed",
      "startTime": "2026-07-22T22:01:39.504754038Z",
      "endTime": "2026-07-22T22:20:44.648330077Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-qcvwjv" for cluster "cluster-candidate-4-20-qcvwjv"
Unexpected error:
    <*fmt.wrapError | 0xc0026a6940>: 
    failed to create NodePool np-qcvwjv: failed waiting for nodepool="np-qcvwjv" for cluster "cluster-candidate-4-20-qcvwjv" in resourcegroup="rg-candidate-4-20-kwkc79fpjrx2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/a9e953c4-caff-45c7-a41d-8606a89637ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/a9e953c4-caff-45c7-a41d-8606a89637ea",
      "name": "a9e953c4-caff-45c7-a41d-8606a89637ea",
      "status": "Failed",
      "startTime": "2026-07-22T22:01:39.504754038Z",
      "endTime": "2026-07-22T22:20:44.648330077Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift
11:45:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0006e58a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-rp6jsc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c",
      "name": "174515fd-ae84-46d9-b491-3d03c175e06c",
      "status": "Failed",
      "startTime": "2026-07-22T12:40:43.885322535Z",
      "endTime": "2026-07-22T12:59:47.104093689Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0006e58a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-rp6jsc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c",
      "name": "174515fd-ae84-46d9-b491-3d03c175e06c",
      "status": "Failed",
      "startTime": "2026-07-22T12:40:43.885322535Z",
      "endTime": "2026-07-22T12:59:47.104093689Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
cluster entered terminal Failed state after role assignment deployment
11:44:47 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail code DenyAssignmentAuthorizationFailed; detail message Access denied by deny assignment for action "Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action".; provider Microsoft.Authorization
08:10:44 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00003c078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xxxldl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
      "name": "9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
      "status": "Failed",
      "startTime": "2026-07-22T08:31:33.476055825Z",
      "endTime": "2026-07-22T08:50:37.393817181Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-8skrs-nxzhn": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-nxzhn": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: DenyAssignmentAuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DenyAssignmentAuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
      }
    }
    --------------------------------------------------------------------------------
    
      - VM "cilium-cluster-cilium-np-8skrs-wwbw4": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-wwbw4": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: DenyAssignmentAuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DenyAssignmentAuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00003c078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xxxldl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
      "name": "9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
      "status": "Failed",
      "startTime": "2026-07-22T08:31:33.476055825Z",
      "endTime": "2026-07-22T08:50:37.393817181Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-8skrs-nxzhn": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-nxzhn": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: DenyAssignmentAuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DenyAssignmentAuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
      }
    }
    --------------------------------------------------------------------------------
    
      - VM "cilium-cluster-cilium-np-8skrs-wwbw4": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-wwbw4": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: DenyAssignmentAuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DenyAssignmentAuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:57:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-79gssd" for cluster "cluster-candidate-4-21-79gssd"
Unexpected error:
    <*fmt.wrapError | 0xc00061b560>: 
    failed to create NodePool np-79gssd: failed waiting for nodepool="np-79gssd" for cluster "cluster-candidate-4-21-79gssd" in resourcegroup="rg-candidate-4-21-rjrdqb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/985baa3e-b3d4-4c63-a507-079bb53207c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/985baa3e-b3d4-4c63-a507-079bb53207c7",
      "name": "985baa3e-b3d4-4c63-a507-079bb53207c7",
      "status": "Failed",
      "startTime": "2026-07-22T06:00:51.385178815Z",
      "endTime": "2026-07-22T06:19:51.057036582Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-79gssd" for cluster "cluster-candidate-4-21-79gssd"
Unexpected error:
    <*fmt.wrapError | 0xc00061b560>: 
    failed to create NodePool np-79gssd: failed waiting for nodepool="np-79gssd" for cluster "cluster-candidate-4-21-79gssd" in resourcegroup="rg-candidate-4-21-rjrdqb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/985baa3e-b3d4-4c63-a507-079bb53207c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/985baa3e-b3d4-4c63-a507-079bb53207c7",
      "name": "985baa3e-b3d4-4c63-a507-079bb53207c7",
      "status": "Failed",
      "startTime": "2026-07-22T06:00:51.385178815Z",
      "endTime": "2026-07-22T06:19:51.057036582Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 3
e2e (3)
cluster entered terminal Failed state after role assignment deployment
02:00:45 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
02:00:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-20-qnfrfst59rxk/cluster-candidate-4-20-z9f7qm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b10b60>: 
    failed to create HCP cluster cluster-candidate-4-20-z9f7qm: failed waiting for cluster="cluster-candidate-4-20-z9f7qm" in resourcegroup="rg-candidate-4-20-qnfrfst59rxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b",
      "name": "952533d9-7ad3-4b91-b8b8-23d93e01328b",
      "status": "Failed",
      "startTime": "2026-07-22T02:05:40.15224731Z",
      "endTime": "2026-07-22T02:24:41.285079074Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-20-qnfrfst59rxk/cluster-candidate-4-20-z9f7qm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b10b60>: 
    failed to create HCP cluster cluster-candidate-4-20-z9f7qm: failed waiting for cluster="cluster-candidate-4-20-z9f7qm" in resourcegroup="rg-candidate-4-20-qnfrfst59rxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b",
      "name": "952533d9-7ad3-4b91-b8b8-23d93e01328b",
      "status": "Failed",
      "startTime": "2026-07-22T02:05:40.15224731Z",
      "endTime": "2026-07-22T02:24:41.285079074Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
02:00:45 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0005871c0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-sj5lp4s8hdhx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75",
      "name": "913f1b53-79a1-4c56-90f3-f94f835b8c75",
      "status": "Failed",
      "startTime": "2026-07-22T02:04:09.655224875Z",
      "endTime": "2026-07-22T02:23:21.611762653Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0005871c0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-sj5lp4s8hdhx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75",
      "name": "913f1b53-79a1-4c56-90f3-f94f835b8c75",
      "status": "Failed",
      "startTime": "2026-07-22T02:04:09.655224875Z",
      "endTime": "2026-07-22T02:23:21.611762653Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
cluster entered terminal Failed state after role assignment deployment
00:53:11 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
00:48:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-v8pbwf" for cluster "cluster-candidate-4-20-v8pbwf"
Unexpected error:
    <*fmt.wrapError | 0xc000fb8780>: 
    failed to create NodePool np-v8pbwf: failed waiting for nodepool="np-v8pbwf" for cluster "cluster-candidate-4-20-v8pbwf" in resourcegroup="rg-candidate-4-20-tr95ln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/4dcd2df8-598c-4ddd-ae99-b9917b436164
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/4dcd2df8-598c-4ddd-ae99-b9917b436164",
      "name": "4dcd2df8-598c-4ddd-ae99-b9917b436164",
      "status": "Failed",
      "startTime": "2026-07-22T01:51:53.026515054Z",
      "endTime": "2026-07-22T02:11:00.801292673Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-v8pbwf" for cluster "cluster-candidate-4-20-v8pbwf"
Unexpected error:
    <*fmt.wrapError | 0xc000fb8780>: 
    failed to create NodePool np-v8pbwf: failed waiting for nodepool="np-v8pbwf" for cluster "cluster-candidate-4-20-v8pbwf" in resourcegroup="rg-candidate-4-20-tr95ln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/4dcd2df8-598c-4ddd-ae99-b9917b436164
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/4dcd2df8-598c-4ddd-ae99-b9917b436164",
      "name": "4dcd2df8-598c-4ddd-ae99-b9917b436164",
      "status": "Failed",
      "startTime": "2026-07-22T01:51:53.026515054Z",
      "endTime": "2026-07-22T02:11:00.801292673Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred