<
Single Day: Oct 21▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 52m 4s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
canadacentral e2e failed n/a 2
e2e (2)
"Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
23:00:10 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 307.279s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0x337cde092010>: 
    E1021 23:30:50.528002    2868 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    {
        s: "E1021 23:30:50.528002    2868 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"",
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 307.279s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0x337cde092010>: 
    E1021 23:30:50.528002    2868 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    {
        s: "E1021 23:30:50.528002    2868 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"",
    }
occurred
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
23:00:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xb243792ee20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0xb243792edc0>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xb243792ee20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0xb243792edc0>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
2h 23m 15s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
centralindia e2e failed n/a 1
e2e (1)
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
22:23:42 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x3866dcc37c20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
      - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        err: <*errors.errorString | 0x3866dc67cd90>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x3866dcc37c20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
      - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        err: <*errors.errorString | 0x3866dc67cd90>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        },
    }
occurred
2h 25m 32s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
australiaeast n/a passed n/a 0 n/a
2h 27m 23s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
switzerlandnorth n/a passed n/a 0 n/a
2h 24m 55s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
canadacentral n/a passed n/a 0 n/a
2h 13m 42s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
uksouth n/a passed n/a 0 n/a
2h 25m 8s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
uksouth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
16:01:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x29388dc21b20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l6ddnj7vhmkq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec",
      "name": "8669d017-4213-4ec1-86fe-07c71e8067ec",
      "status": "Failed",
      "startTime": "2026-10-21T16:34:55.668662338Z",
      "endTime": "2026-10-21T16:53:59.759177283Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
            correlation ID: 7627942b-6e2a-4d1f-a2c9-7a7ff2104478
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x29388dc21b20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l6ddnj7vhmkq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec",
      "name": "8669d017-4213-4ec1-86fe-07c71e8067ec",
      "status": "Failed",
      "startTime": "2026-10-21T16:34:55.668662338Z",
      "endTime": "2026-10-21T16:53:59.759177283Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
            correlation ID: 7627942b-6e2a-4d1f-a2c9-7a7ff2104478
            client request ID: unavailable
occurred
2h 21m 12s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
eastus2 n/a passed n/a 0 n/a
2h 23m 5s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
brazilsouth n/a passed n/a 0 n/a
2h 28m 34s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westeurope n/a passed n/a 0 n/a
2h 10m 59s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
australiaeast n/a passed n/a 0 n/a
2h 41m 13s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
australiaeast n/a passed n/a 0 n/a