No runs match your search.
Environment: PROD
| Time (UTC) | Runtime | Job | Region | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|---|---|
| 1h 52m 4s | periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly post-goodmerged PR |
canadacentral | e2e | failed | n/a | 2 | e2e (2)"Couldn't get current server API group list" err="the server has asked for the client to provide credentials" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 307.279s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0x337cde092010>:
E1021 23:30:50.528002 2868 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
{
s: "E1021 23:30:50.528002 2868 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 307.279s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0x337cde092010>:
E1021 23:30:50.528002 2868 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
{
s: "E1021 23:30:50.528002 2868 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"",
}
occurredVerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
<*fmt.wrapError | 0xb243792ee20>:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
{
msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <*fmt.wrapError | 0xb243792edc0>{
msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
<*fmt.wrapError | 0xb243792ee20>:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
{
msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <*fmt.wrapError | 0xb243792edc0>{
msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred |
|
| 2h 23m 15s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
centralindia | e2e | failed | n/a | 1 | e2e (1)VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors: Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
<*fmt.wrapError | 0x3866dcc37c20>:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
- pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
{
msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
err: <*errors.errorString | 0x3866dc67cd90>{
s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
<*fmt.wrapError | 0x3866dcc37c20>:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
- pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
{
msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.005s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
err: <*errors.errorString | 0x3866dc67cd90>{
s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
},
}
occurred |
|
| 2h 25m 32s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
australiaeast | n/a | passed | n/a | 0 | n/a | |
| 2h 27m 23s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
switzerlandnorth | n/a | passed | n/a | 0 | n/a | |
| 2h 24m 55s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
canadacentral | n/a | passed | n/a | 0 | n/a | |
| 2h 13m 42s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
uksouth | n/a | passed | n/a | 0 | n/a | |
| 2h 25m 8s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
uksouth | e2e | failed | n/a | 1 | e2e (1)ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x29388dc21b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l6ddnj7vhmkq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec",
"name": "8669d017-4213-4ec1-86fe-07c71e8067ec",
"status": "Failed",
"startTime": "2026-10-21T16:34:55.668662338Z",
"endTime": "2026-10-21T16:53:59.759177283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
correlation ID: 7627942b-6e2a-4d1f-a2c9-7a7ff2104478
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x29388dc21b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l6ddnj7vhmkq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec",
"name": "8669d017-4213-4ec1-86fe-07c71e8067ec",
"status": "Failed",
"startTime": "2026-10-21T16:34:55.668662338Z",
"endTime": "2026-10-21T16:53:59.759177283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
correlation ID: 7627942b-6e2a-4d1f-a2c9-7a7ff2104478
client request ID: unavailable
occurred |
|
| 2h 21m 12s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
eastus2 | n/a | passed | n/a | 0 | n/a | |
| 2h 23m 5s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
brazilsouth | n/a | passed | n/a | 0 | n/a | |
| 2h 28m 34s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
westeurope | n/a | passed | n/a | 0 | n/a | |
| 2h 10m 59s | periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
australiaeast | n/a | passed | n/a | 0 | n/a | |
| 2h 41m 13s | branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR |
australiaeast | n/a | passed | n/a | 0 | n/a |