Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 72 | 68.06% (off track) | -26.94pp | +9.79pp | 91.67% (66/72) | +8.33pp | 74.24% (49/66) | +3.77pp |
| INT | 95% - All E2E job runs | 21 | 66.67% (off track) | -28.33pp | -4.76pp | n/a | n/a | 82.35% (14/17) | -17.65pp |
| STG | 95% - All E2E job runs | 8 | 62.50% (off track) | -32.50pp | -21.50pp | n/a | n/a | 62.50% (5/8) | -25.00pp |
| PROD | 95% - All E2E job runs | 27 | 51.85% (off track) | -43.15pp | +5.42pp | n/a | n/a | 51.85% (14/27) | +5.42pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 6 | 3.33%6 of 180 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 · Nov 18: 5 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-11-18T21:52:29Z Ended: 2026-11-18T22:04:31Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8598400-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-11-18T21:52:29Z Ended: 2026-11-18T22:04:31Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8598400-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-18T18:41:14Z Ended: 2026-11-18T18:49:13Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2898432-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-18T14:39:04Z Ended: 2026-11-18T14:49:07Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8171776-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (6) | |||||||||||||||||||||||||||||||
ERROR CODE: 429; provider Microsoft.ManagedIdentityfull failure pattern: ERROR CODE: 429; provider Microsoft.ManagedIdentity Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | e2e | 5 | 2.78%5 of 180 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | Show trend detailsNov 12: 1 · Nov 13: 1 · Nov 14: 1 · Nov 15: 0 · Nov 16: 2 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x35ebe6445d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: 3e221ae8-288f-419c-b94d-d5119240e2c2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x35ebe6445d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: 3e221ae8-288f-419c-b94d-d5119240e2c2
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x303d4ee7620>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: c7391ef9-779f-4fec-a1fc-92e1977eada5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x303d4ee7620>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: c7391ef9-779f-4fec-a1fc-92e1977eada5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0xc427f5a17d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: bbc0aa39-601e-43c7-8d35-95370cb55622
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0xc427f5a17d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: bbc0aa39-601e-43c7-8d35-95370cb55622
client request ID: unavailable
occurredContributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 5 | 2.78%5 of 180 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 2 · Nov 18: 2 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-f8kpfw"
Unexpected error:
<*errors.joinError | 0x1fe1de7a9380>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-f8kpfw-z58tq-s8k8p egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-f8kpfw-z58tq-s8k8p": nodes "cluster-np-f8kpfw-z58tq-s8k8p" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1fe1ddcbb560>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-f8kpfw-z58tq-s8k8p egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-f8kpfw-z58tq-s8k8p\": nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1fe1ddcbb540>{
msg: "failed to label node cluster-np-f8kpfw-z58tq-s8k8p egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-f8kpfw-z58tq-s8k8p\": nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1fe1ddcbb520>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-f8kpfw-z58tq-s8k8p\": nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1fe1de5c7220>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-f8kpfw-z58tq-s8k8p",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-f8kpfw"
Unexpected error:
<*errors.joinError | 0x1fe1de7a9380>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-f8kpfw-z58tq-s8k8p egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-f8kpfw-z58tq-s8k8p": nodes "cluster-np-f8kpfw-z58tq-s8k8p" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1fe1ddcbb560>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-f8kpfw-z58tq-s8k8p egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-f8kpfw-z58tq-s8k8p\": nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1fe1ddcbb540>{
msg: "failed to label node cluster-np-f8kpfw-z58tq-s8k8p egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-f8kpfw-z58tq-s8k8p\": nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1fe1ddcbb520>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-f8kpfw-z58tq-s8k8p\": nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1fe1de5c7220>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-f8kpfw-z58tq-s8k8p\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-f8kpfw-z58tq-s8k8p",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-ww9rlr"
Unexpected error:
<*errors.joinError | 0x11da19f20648>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ww9rlr-97ppl-2jx82 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-ww9rlr-97ppl-2jx82": nodes "cluster-np-ww9rlr-97ppl-2jx82" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x11da1a14d520>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ww9rlr-97ppl-2jx82 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ww9rlr-97ppl-2jx82\": nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x11da1a14d500>{
msg: "failed to label node cluster-np-ww9rlr-97ppl-2jx82 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ww9rlr-97ppl-2jx82\": nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x11da1a14d4e0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ww9rlr-97ppl-2jx82\": nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x11da1a1475e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-ww9rlr-97ppl-2jx82",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-ww9rlr"
Unexpected error:
<*errors.joinError | 0x11da19f20648>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ww9rlr-97ppl-2jx82 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-ww9rlr-97ppl-2jx82": nodes "cluster-np-ww9rlr-97ppl-2jx82" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x11da1a14d520>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ww9rlr-97ppl-2jx82 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ww9rlr-97ppl-2jx82\": nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x11da1a14d500>{
msg: "failed to label node cluster-np-ww9rlr-97ppl-2jx82 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ww9rlr-97ppl-2jx82\": nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x11da1a14d4e0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ww9rlr-97ppl-2jx82\": nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x11da1a1475e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-ww9rlr-97ppl-2jx82\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-ww9rlr-97ppl-2jx82",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-5-0-6zvmlq"
Unexpected error:
<*errors.joinError | 0x1d2b866e4b88>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-6zvmlq-z7xn5-n8zhd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-6zvmlq-z7xn5-n8zhd": nodes "cluster-np-6zvmlq-z7xn5-n8zhd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1d2b85351a00>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-6zvmlq-z7xn5-n8zhd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-6zvmlq-z7xn5-n8zhd\": nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1d2b853519e0>{
msg: "failed to label node cluster-np-6zvmlq-z7xn5-n8zhd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-6zvmlq-z7xn5-n8zhd\": nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1d2b853519c0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-6zvmlq-z7xn5-n8zhd\": nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1d2b857aec80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-6zvmlq-z7xn5-n8zhd",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-5-0-6zvmlq"
Unexpected error:
<*errors.joinError | 0x1d2b866e4b88>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-6zvmlq-z7xn5-n8zhd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-6zvmlq-z7xn5-n8zhd": nodes "cluster-np-6zvmlq-z7xn5-n8zhd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1d2b85351a00>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-6zvmlq-z7xn5-n8zhd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-6zvmlq-z7xn5-n8zhd\": nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1d2b853519e0>{
msg: "failed to label node cluster-np-6zvmlq-z7xn5-n8zhd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-6zvmlq-z7xn5-n8zhd\": nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1d2b853519c0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-6zvmlq-z7xn5-n8zhd\": nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1d2b857aec80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-6zvmlq-z7xn5-n8zhd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-6zvmlq-z7xn5-n8zhd",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (5) | |||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 4 | 2.22%4 of 180 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 12: 0 · Nov 13: 2 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-89vkxf64tg5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x27b6689ca280>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-89vkxf64tg5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j7442176-mgmt-1/ocm-arohcpci01-2u58h6jvg48mp125p38dvhl3nmhm5ub4-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-89vkxf64tg5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j7442176-mgmt-1/ocm-arohcpci01-2u58h6jvg48mp125p38dvhl3nmhm5ub4-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x27b668b56230>{
s: "dev/westus3/ci01-j7442176-mgmt-1/ocm-arohcpci01-2u58h6jvg48mp125p38dvhl3nmhm5ub4-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-89vkxf64tg5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x27b6689ca280>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-89vkxf64tg5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j7442176-mgmt-1/ocm-arohcpci01-2u58h6jvg48mp125p38dvhl3nmhm5ub4-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-89vkxf64tg5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j7442176-mgmt-1/ocm-arohcpci01-2u58h6jvg48mp125p38dvhl3nmhm5ub4-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x27b668b56230>{
s: "dev/westus3/ci01-j7442176-mgmt-1/ocm-arohcpci01-2u58h6jvg48mp125p38dvhl3nmhm5ub4-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-bmhs6kqst2fs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x29cfb959a000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-bmhs6kqst2fs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j6434560-mgmt-2/ocm-arohcpci01-2u4klgnpjc6dkdujp8fm8jpgjr6ovrtu-cluster router=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000: expected matching summary after test start and within 8m; summaryUID=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000 pass=e10b693e-7cb7-403e-b563-562b1f8896ea at=2026-11-17T00:11:26Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-bmhs6kqst2fs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j6434560-mgmt-2/ocm-arohcpci01-2u4klgnpjc6dkdujp8fm8jpgjr6ovrtu-cluster router=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000: expected matching summary after test start and within 8m; summaryUID=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000 pass=e10b693e-7cb7-403e-b563-562b1f8896ea at=2026-11-17T00:11:26Z",
err: <*errors.errorString | 0x29cfba6ef860>{
s: "dev/centralus/ci01-j6434560-mgmt-2/ocm-arohcpci01-2u4klgnpjc6dkdujp8fm8jpgjr6ovrtu-cluster router=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000: expected matching summary after test start and within 8m; summaryUID=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000 pass=e10b693e-7cb7-403e-b563-562b1f8896ea at=2026-11-17T00:11:26Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-bmhs6kqst2fs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x29cfb959a000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-bmhs6kqst2fs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j6434560-mgmt-2/ocm-arohcpci01-2u4klgnpjc6dkdujp8fm8jpgjr6ovrtu-cluster router=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000: expected matching summary after test start and within 8m; summaryUID=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000 pass=e10b693e-7cb7-403e-b563-562b1f8896ea at=2026-11-17T00:11:26Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-bmhs6kqst2fs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j6434560-mgmt-2/ocm-arohcpci01-2u4klgnpjc6dkdujp8fm8jpgjr6ovrtu-cluster router=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000: expected matching summary after test start and within 8m; summaryUID=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000 pass=e10b693e-7cb7-403e-b563-562b1f8896ea at=2026-11-17T00:11:26Z",
err: <*errors.errorString | 0x29cfba6ef860>{
s: "dev/centralus/ci01-j6434560-mgmt-2/ocm-arohcpci01-2u4klgnpjc6dkdujp8fm8jpgjr6ovrtu-cluster router=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000: expected matching summary after test start and within 8m; summaryUID=1953fdae-41cc-4327-8621-9f3da9545c6d node=aks-userswft1-11654204-vmss000000 pass=e10b693e-7cb7-403e-b563-562b1f8896ea at=2026-11-17T00:11:26Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2079e6d0e3e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
err: <*verifiers.pollObservationError | 0x2079e6d0e360>{
observation: <*errors.errorString | 0x2079e699e7c0>{
s: "dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
},
interruption: <*fmt.wrapError | 0x2079e6d0e340>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2079e6b8e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2079e6d0e300>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2079e80a8180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2079e6d0e3e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
err: <*verifiers.pollObservationError | 0x2079e6d0e360>{
observation: <*errors.errorString | 0x2079e699e7c0>{
s: "dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
},
interruption: <*fmt.wrapError | 0x2079e6d0e340>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2079e6b8e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2079e6d0e300>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2079e80a8180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 4 | 2.22%4 of 180 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 12: 1 · Nov 13: 2 · Nov 14: 2 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-15T00:38:18Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8647040-mgmt-1", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-ntrqw", region="centralus", severity="warning" Description: Pod swift-recorder/swift-recorder-ntrqw has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-14T15:20:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0519808-svc", component="kubernetes-infrastructure", namespace="aks-command", pod="command-1bae86b83ff941b29f7e8e234ffad7cd", region="westus3", severity="warning" Description: Pod aks-command/command-1bae86b83ff941b29f7e8e234ffad7cd has been in a non-ready state for longer than 5 minutes. alert fired 15 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-wdpfr", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-wdpfr has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000001-hj882", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000001-hj882 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-5cffdb78dc-tz654", region="canadacentral", severity="warning" Description: Pod kube-system/hubble-relay-5cffdb78dc-tz654 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000002-5w7z2", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000002-5w7z2 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-cxdpf", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-cxdpf has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-vxj68", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-vxj68 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000000-84std", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000000-84std has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-t6cxk", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-t6cxk has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-sqzqm", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-sqzqm has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-gb9rv", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-gb9rv has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-78789f646f-2wdlw", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-78789f646f-2wdlw has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-78789f646f-z5ntg", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-78789f646f-z5ntg has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-wr8wf", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-wr8wf has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-zncb7", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-zncb7 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-11-13T20:44:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-9vgxw", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-9vgxw has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | alert | 4 | 2.22%4 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsNov 12: 2 · Nov 13: 4 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-14T00:10:33Z Ended: 2026-11-14T00:16:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5892608-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2l7k5ong7lhsdb3rmd7e6f9npoppbg-cluster", pod="openshift-apiserver-65cdb567cb-vrzcv", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2u2l7k5ong7lhsdb3rmd7e6f9npoppbg-cluster/openshift-apiserver-65cdb567cb-vrzcv has been in a non-ready state for longer than 5 minutes. alert fired 32 time(s) (14 unknown, 18 known) Firing 1: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster", pod="kube-apiserver-578576bffb-f6whx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster/kube-apiserver-578576bffb-f6whx has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:41:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="router-c76569fc9-6cfvc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/router-c76569fc9-6cfvc has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:41:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpes06vummjujuaun3cf6766c5d4-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpes06vummjujuaun3cf6766c5d4-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-11-13T22:41:33Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvd6vuver3peg6br0ldulhn5em1d2-cluster", pod="kube-state-metrics-hcp-867779fc97-5kqgw", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvd6vuver3peg6br0ldulhn5em1d2-cluster/kube-state-metrics-hcp-867779fc97-5kqgw has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:49:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster", pod="openshift-apiserver-548798444f-fkvbc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster/openshift-apiserver-548798444f-fkvbc has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster", pod="openshift-apiserver-7db68f8b9c-ztk8f", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster/openshift-apiserver-7db68f8b9c-ztk8f has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2k00np1fd7kopst9snbjsh5ghmkl25-cluster", pod="kube-state-metrics-hcp-867779fc97-4rb5d", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2k00np1fd7kopst9snbjsh5ghmkl25-cluster/kube-state-metrics-hcp-867779fc97-4rb5d has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster", pod="kube-controller-manager-7b9955c77b-gt9w8", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster/kube-controller-manager-7b9955c77b-gt9w8 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2k00tuj5acf49s1c11mi5h9ldau5ps-cluster", pod="openshift-apiserver-69b9c969cf-2bfxx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2k00tuj5acf49s1c11mi5h9ldau5ps-cluster/openshift-apiserver-69b9c969cf-2bfxx has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="kube-state-metrics-hcp-867779fc97-nnqft", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/kube-state-metrics-hcp-867779fc97-nnqft has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-11-13T22:46:32Z Ended: 2026-11-13T22:52:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juvn997fddu0ckuf17ouet8eoim9t-cluster", pod="openshift-apiserver-f95d96f84-p58r8", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juvn997fddu0ckuf17ouet8eoim9t-cluster/openshift-apiserver-f95d96f84-p58r8 has been in a non-ready state for longer than 5 minutes. alert fired 14 time(s) (1 unknown, 13 known) Firing 1: State: Resolved Started: 2026-11-13T13:12:50Z Ended: 2026-11-13T13:28:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5993600-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster", pod="etcd-1", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 3 | 1.67%3 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 2 · Nov 18: 1 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mp2mtkll2jtz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x187a66b56a60>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mp2mtkll2jtz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j9895424-mgmt-1/ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster router=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002: expected matching summary after test start and within 8m; summaryUID=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002 pass=e3a8050b-8385-4bca-a682-d931aa14ab37 at=2026-11-18T03:59:39Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mp2mtkll2jtz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j9895424-mgmt-1/ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster router=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002: expected matching summary after test start and within 8m; summaryUID=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002 pass=e3a8050b-8385-4bca-a682-d931aa14ab37 at=2026-11-18T03:59:39Z",
err: <*verifiers.pollObservationError | 0x187a66b56a40>{
observation: <*errors.errorString | 0x187a669cfaf0>{
s: "dev/westus3/ci01-j9895424-mgmt-1/ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster router=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002: expected matching summary after test start and within 8m; summaryUID=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002 pass=e3a8050b-8385-4bca-a682-d931aa14ab37 at=2026-11-18T03:59:39Z",
},
interruption: <*fmt.wrapError | 0x187a66b56a20>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9895424-mgmt-1\"; let ns = \"ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster\"; let start = datetime(2026-11-18T03:45:11.8010088Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x187a673307d0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x187a66b569e0>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9895424-mgmt-1\"; let ns = \"ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster\"; let start = datetime(2026-11-18T03:45:11.8010088Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x187a68399740>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mp2mtkll2jtz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x187a66b56a60>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mp2mtkll2jtz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j9895424-mgmt-1/ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster router=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002: expected matching summary after test start and within 8m; summaryUID=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002 pass=e3a8050b-8385-4bca-a682-d931aa14ab37 at=2026-11-18T03:59:39Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mp2mtkll2jtz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j9895424-mgmt-1/ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster router=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002: expected matching summary after test start and within 8m; summaryUID=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002 pass=e3a8050b-8385-4bca-a682-d931aa14ab37 at=2026-11-18T03:59:39Z",
err: <*verifiers.pollObservationError | 0x187a66b56a40>{
observation: <*errors.errorString | 0x187a669cfaf0>{
s: "dev/westus3/ci01-j9895424-mgmt-1/ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster router=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002: expected matching summary after test start and within 8m; summaryUID=50441035-8cff-48f6-bdd7-44a228731983 node=aks-userswft3-70577556-vmss000002 pass=e3a8050b-8385-4bca-a682-d931aa14ab37 at=2026-11-18T03:59:39Z",
},
interruption: <*fmt.wrapError | 0x187a66b56a20>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9895424-mgmt-1\"; let ns = \"ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster\"; let start = datetime(2026-11-18T03:45:11.8010088Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x187a673307d0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x187a66b569e0>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9895424-mgmt-1\"; let ns = \"ocm-arohcpci01-2u5d35cipav505mljtttdk6d6h3qp04q-cluster\"; let start = datetime(2026-11-18T03:45:11.8010088Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x187a68399740>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-kv4jbjbk5vck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x347f21f96300>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-kv4jbjbk5vck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0987392-mgmt-1/ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster router=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003: expected matching summary after test start and within 8m; summaryUID=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003 pass=5490c58d-8d98-41de-8126-a29f3d335d5a at=2026-11-17T17:01:52Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-kv4jbjbk5vck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0987392-mgmt-1/ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster router=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003: expected matching summary after test start and within 8m; summaryUID=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003 pass=5490c58d-8d98-41de-8126-a29f3d335d5a at=2026-11-17T17:01:52Z",
err: <*verifiers.pollObservationError | 0x347f21f96100>{
observation: <*errors.errorString | 0x347f23d02320>{
s: "dev/centralus/ci01-j0987392-mgmt-1/ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster router=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003: expected matching summary after test start and within 8m; summaryUID=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003 pass=5490c58d-8d98-41de-8126-a29f3d335d5a at=2026-11-17T17:01:52Z",
},
interruption: <*fmt.wrapError | 0x347f21f960e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0987392-mgmt-1\"; let ns = \"ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster\"; let start = datetime(2026-11-17T16:46:21.9485701Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x347f2265c000>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x347f21f96000>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0987392-mgmt-1\"; let ns = \"ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster\"; let start = datetime(2026-11-17T16:46:21.9485701Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x347f238e6000>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-kv4jbjbk5vck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x347f21f96300>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-kv4jbjbk5vck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0987392-mgmt-1/ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster router=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003: expected matching summary after test start and within 8m; summaryUID=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003 pass=5490c58d-8d98-41de-8126-a29f3d335d5a at=2026-11-17T17:01:52Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-kv4jbjbk5vck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0987392-mgmt-1/ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster router=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003: expected matching summary after test start and within 8m; summaryUID=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003 pass=5490c58d-8d98-41de-8126-a29f3d335d5a at=2026-11-17T17:01:52Z",
err: <*verifiers.pollObservationError | 0x347f21f96100>{
observation: <*errors.errorString | 0x347f23d02320>{
s: "dev/centralus/ci01-j0987392-mgmt-1/ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster router=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003: expected matching summary after test start and within 8m; summaryUID=31b7c623-e369-4205-a7c5-4c17bd6657d0 node=aks-userswft1-33714086-vmss000003 pass=5490c58d-8d98-41de-8126-a29f3d335d5a at=2026-11-17T17:01:52Z",
},
interruption: <*fmt.wrapError | 0x347f21f960e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0987392-mgmt-1\"; let ns = \"ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster\"; let start = datetime(2026-11-17T16:46:21.9485701Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x347f2265c000>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x347f21f96000>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0987392-mgmt-1\"; let ns = \"ocm-arohcpci01-2u53eqq46ot9tcn1dko0s8vfi3tagpfk-cluster\"; let start = datetime(2026-11-17T16:46:21.9485701Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x347f238e6000>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qh92kg9qptkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x11852edfac60>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qh92kg9qptkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j4759168-mgmt-2/ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qh92kg9qptkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j4759168-mgmt-2/ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x11852edfac20>{
observation: <*errors.errorString | 0x11852ee366a0>{
s: "dev/westus3/ci01-j4759168-mgmt-2/ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x11852edfabc0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j4759168-mgmt-2\"; let ns = \"ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster\"; let start = datetime(2026-11-17T02:00:45.2425659Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x11852eb82320>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x11852edfab80>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j4759168-mgmt-2\"; let ns = \"ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster\"; let start = datetime(2026-11-17T02:00:45.2425659Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1185310871a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qh92kg9qptkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x11852edfac60>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qh92kg9qptkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j4759168-mgmt-2/ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qh92kg9qptkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/westus3/ci01-j4759168-mgmt-2/ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x11852edfac20>{
observation: <*errors.errorString | 0x11852ee366a0>{
s: "dev/westus3/ci01-j4759168-mgmt-2/ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x11852edfabc0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j4759168-mgmt-2\"; let ns = \"ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster\"; let start = datetime(2026-11-17T02:00:45.2425659Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x11852eb82320>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x11852edfab80>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j4759168-mgmt-2\"; let ns = \"ocm-arohcpci01-2u4mf5mp3gkbvm9l6v27l0dpmnga4ri9-cluster\"; let start = datetime(2026-11-17T02:00:45.2425659Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1185310871a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 2 | 1.11%2 of 180 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 12: 1 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x4f2057fa1c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
err: <*verifiers.pollObservationError | 0x4f2057fa180>{
observation: <*errors.errorString | 0x4f2037c8bd0>{
s: "dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
},
interruption: <*fmt.wrapError | 0x4f2057fa160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x4f203456140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x4f2057fa140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x4f205896180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x4f2057fa1c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
err: <*verifiers.pollObservationError | 0x4f2057fa180>{
observation: <*errors.errorString | 0x4f2037c8bd0>{
s: "dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
},
interruption: <*fmt.wrapError | 0x4f2057fa160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x4f203456140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x4f2057fa140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x4f205896180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2387254f8000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x238726e308c0>{
s: "dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2387254f8000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x238726e308c0>{
s: "dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||
alert [svc] BackendControllerRetryHotLoop firedfull failure pattern: alert [svc] BackendControllerRetryHotLoop fired Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | alert | 2 | 1.11%2 of 180 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 12: 1 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)alert fired 6 time(s) Firing 1: State: Resolved Started: 2026-11-16T17:27:24Z Ended: 2026-11-16T17:59:21Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="placement", region="westus3", severity="warning" Description: Backend controller workqueue placement has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 2: State: Resolved Started: 2026-11-16T17:30:23Z Ended: 2026-11-16T17:58:24Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="identityroleassignments", region="westus3", severity="warning" Description: Backend controller workqueue identityroleassignments has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 3: State: Resolved Started: 2026-11-16T18:06:26Z Ended: 2026-11-16T18:19:21Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="ensuremanagedresourcegroup", region="westus3", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 4: State: Resolved Started: 2026-11-16T18:23:23Z Ended: 2026-11-16T18:52:22Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="placement", region="westus3", severity="warning" Description: Backend controller workqueue placement has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 5: State: Resolved Started: 2026-11-16T18:37:23Z Ended: 2026-11-16T18:49:25Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="identityroleassignments", region="westus3", severity="warning" Description: Backend controller workqueue identityroleassignments has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 6: State: Fired (not resolved) Started: 2026-11-16T18:57:25Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="ensuremanagedresourcegroup", region="westus3", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-14T13:35:08Z Ended: 2026-11-14T13:49:00Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j7717504-svc", component="backend", name="ensuremanagedresourcegroup", region="centralus", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 2 | 1.11%2 of 180 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 12: 1 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ndnmxlhfl2dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3004ceab61c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ndnmxlhfl2dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/canadacentral/ci01-j4606336-mgmt-2/ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster router=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001: expected matching summary after test start and within 8m; summaryUID=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001 pass=5b902719-1287-4353-bfe1-5807d2f380c6 at=2026-11-17T16:05:50Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ndnmxlhfl2dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/canadacentral/ci01-j4606336-mgmt-2/ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster router=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001: expected matching summary after test start and within 8m; summaryUID=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001 pass=5b902719-1287-4353-bfe1-5807d2f380c6 at=2026-11-17T16:05:50Z",
err: <*verifiers.pollObservationError | 0x3004ceab61a0>{
observation: <*errors.errorString | 0x3004d0723030>{
s: "dev/canadacentral/ci01-j4606336-mgmt-2/ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster router=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001: expected matching summary after test start and within 8m; summaryUID=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001 pass=5b902719-1287-4353-bfe1-5807d2f380c6 at=2026-11-17T16:05:50Z",
},
interruption: <*fmt.wrapError | 0x3004ceab6180>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4606336-mgmt-2\"; let ns = \"ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster\"; let start = datetime(2026-11-17T15:50:10.1165377Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3004ce7502d0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3004ceab6160>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4606336-mgmt-2\"; let ns = \"ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster\"; let start = datetime(2026-11-17T15:50:10.1165377Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3004cfffa0c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ndnmxlhfl2dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3004ceab61c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ndnmxlhfl2dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/canadacentral/ci01-j4606336-mgmt-2/ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster router=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001: expected matching summary after test start and within 8m; summaryUID=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001 pass=5b902719-1287-4353-bfe1-5807d2f380c6 at=2026-11-17T16:05:50Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ndnmxlhfl2dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/canadacentral/ci01-j4606336-mgmt-2/ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster router=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001: expected matching summary after test start and within 8m; summaryUID=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001 pass=5b902719-1287-4353-bfe1-5807d2f380c6 at=2026-11-17T16:05:50Z",
err: <*verifiers.pollObservationError | 0x3004ceab61a0>{
observation: <*errors.errorString | 0x3004d0723030>{
s: "dev/canadacentral/ci01-j4606336-mgmt-2/ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster router=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001: expected matching summary after test start and within 8m; summaryUID=c8ba6fa2-1c5f-4a8f-916b-16d25081906f node=aks-userswft3-18527780-vmss000001 pass=5b902719-1287-4353-bfe1-5807d2f380c6 at=2026-11-17T16:05:50Z",
},
interruption: <*fmt.wrapError | 0x3004ceab6180>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4606336-mgmt-2\"; let ns = \"ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster\"; let start = datetime(2026-11-17T15:50:10.1165377Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3004ce7502d0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3004ceab6160>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4606336-mgmt-2\"; let ns = \"ocm-arohcpci01-2u52k7h10qp804mgs8n7kndubi15ib7o-cluster\"; let start = datetime(2026-11-17T15:50:10.1165377Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3004cfffa0c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2af5093ab1a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x2af5093ab180>{
observation: <*errors.errorString | 0x2af508c1bd90>{
s: "dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x2af5093ab160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2af508aa2820>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2af5093ab140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2af509e57260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2af5093ab1a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x2af5093ab180>{
observation: <*errors.errorString | 0x2af508c1bd90>{
s: "dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x2af5093ab160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2af508aa2820>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2af5093ab140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2af509e57260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 1.11%2 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 2 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | PROD | |||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:210]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x22c18c6b24a0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-5hrn9mspwfjd" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12",
"name": "83b4c400-775d-4ffd-9b67-6287332ebc12",
"status": "Failed",
"startTime": "2026-11-14T13:07:18.872762914Z",
"endTime": "2026-11-14T13:36:22.180780509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
correlation ID: 8b44e4e2-2314-4db7-8d11-8d49e51bd3b0
client request ID: 07dcbf36-1afc-4c8f-8c6d-97deadcb3181
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:210]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x22c18c6b24a0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-5hrn9mspwfjd" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12",
"name": "83b4c400-775d-4ffd-9b67-6287332ebc12",
"status": "Failed",
"startTime": "2026-11-14T13:07:18.872762914Z",
"endTime": "2026-11-14T13:36:22.180780509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
correlation ID: 8b44e4e2-2314-4db7-8d11-8d49e51bd3b0
client request ID: 07dcbf36-1afc-4c8f-8c6d-97deadcb3181
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:81]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x2c409df90980>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-q8h7nh5ldb6w" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc",
"name": "6302990a-b328-4ae4-97f7-38baacded1bc",
"status": "Failed",
"startTime": "2026-11-14T13:04:36.402299697Z",
"endTime": "2026-11-14T13:33:42.521828312Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
correlation ID: 1feffd8c-99db-457e-ada3-5a60ca87a066
client request ID: 124c0023-a5a5-4e28-9aad-bd6fd1f2656a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:81]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x2c409df90980>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-q8h7nh5ldb6w" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc",
"name": "6302990a-b328-4ae4-97f7-38baacded1bc",
"status": "Failed",
"startTime": "2026-11-14T13:04:36.402299697Z",
"endTime": "2026-11-14T13:33:42.521828312Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
correlation ID: 1feffd8c-99db-457e-ada3-5a60ca87a066
client request ID: 124c0023-a5a5-4e28-9aad-bd6fd1f2656a
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:293]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x1f91d1e347c0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-qdvhlcv5hhb9" to finish creating: GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a",
"name": "b909bf18-c4a9-46c0-b023-9eed9656600a",
"status": "Failed",
"startTime": "2026-11-13T19:30:06.636680304Z",
"endTime": "2026-11-13T19:59:07.03404368Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
correlation ID: 7f002ae4-4687-4f1e-a00b-6530f9b7a9f9
client request ID: 9d864843-a38b-4d44-be3f-c3a6e910cb19
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:293]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x1f91d1e347c0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-qdvhlcv5hhb9" to finish creating: GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a",
"name": "b909bf18-c4a9-46c0-b023-9eed9656600a",
"status": "Failed",
"startTime": "2026-11-13T19:30:06.636680304Z",
"endTime": "2026-11-13T19:59:07.03404368Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
correlation ID: 7f002ae4-4687-4f1e-a00b-6530f9b7a9f9
client request ID: 9d864843-a38b-4d44-be3f-c3a6e910cb19
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Mi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 1.11%2 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 2 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x2081b7d14060>:
resource group "np-autoscaling-d9nbfctr94mh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-d9nbfctr94mh" to finish deleting: GET https://rp.j9576960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1be1eff5-6282-418d-84da-bf356c954b60
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1be1eff5-6282-418d-84da-bf356c954b60",
"name": "1be1eff5-6282-418d-84da-bf356c954b60",
"status": "Failed",
"startTime": "2026-11-18T01:28:33.540815554Z",
"endTime": "2026-11-18T01:52:37.539908927Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9576960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1be1eff5-6282-418d-84da-bf356c954b60
correlation ID: c57b140a-01d7-4e2f-a7e2-4cbff713f83f
client request ID: bd5a09e4-85f5-4727-bbdc-51e3ecbe392c
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x2081b7d14060>:
resource group "np-autoscaling-d9nbfctr94mh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-d9nbfctr94mh" to finish deleting: GET https://rp.j9576960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1be1eff5-6282-418d-84da-bf356c954b60
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1be1eff5-6282-418d-84da-bf356c954b60",
"name": "1be1eff5-6282-418d-84da-bf356c954b60",
"status": "Failed",
"startTime": "2026-11-18T01:28:33.540815554Z",
"endTime": "2026-11-18T01:52:37.539908927Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9576960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1be1eff5-6282-418d-84da-bf356c954b60
correlation ID: c57b140a-01d7-4e2f-a7e2-4cbff713f83f
client request ID: bd5a09e4-85f5-4727-bbdc-51e3ecbe392c
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x12de4900a210>:
resource group "ephemeral-osdisk-vl25skc7p49s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-vl25skc7p49s" to finish deleting: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2bf16b6-8fac-4799-ad05-f0795f94e579
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2bf16b6-8fac-4799-ad05-f0795f94e579",
"name": "b2bf16b6-8fac-4799-ad05-f0795f94e579",
"status": "Failed",
"startTime": "2026-11-14T13:18:15.617279757Z",
"endTime": "2026-11-14T13:42:22.806870312Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2bf16b6-8fac-4799-ad05-f0795f94e579
correlation ID: d6891843-0a67-4c46-80bd-2bbdca59c2d3
client request ID: bfdf6458-1f5c-4528-a60f-0b678456b2e5
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x12de4900a210>:
resource group "ephemeral-osdisk-vl25skc7p49s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-vl25skc7p49s" to finish deleting: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2bf16b6-8fac-4799-ad05-f0795f94e579
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2bf16b6-8fac-4799-ad05-f0795f94e579",
"name": "b2bf16b6-8fac-4799-ad05-f0795f94e579",
"status": "Failed",
"startTime": "2026-11-14T13:18:15.617279757Z",
"endTime": "2026-11-14T13:42:22.806870312Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2bf16b6-8fac-4799-ad05-f0795f94e579
correlation ID: d6891843-0a67-4c46-80bd-2bbdca59c2d3
client request ID: bfdf6458-1f5c-4528-a60f-0b678456b2e5
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x3366ec2b20f0>:
resource group "pullsecret-test-8vrs9v6mj4c5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-8vrs9v6mj4c5" to finish deleting: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e9734814-276e-4a55-82bc-7fe55e4a8cd1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e9734814-276e-4a55-82bc-7fe55e4a8cd1",
"name": "e9734814-276e-4a55-82bc-7fe55e4a8cd1",
"status": "Failed",
"startTime": "2026-11-14T13:19:30.200005834Z",
"endTime": "2026-11-14T13:43:33.470145925Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e9734814-276e-4a55-82bc-7fe55e4a8cd1
correlation ID: 3303484d-5e22-4995-a1a3-92df5a0d563f
client request ID: 303745a2-8e9f-4e2f-96c3-31fdb0e1330e
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x3366ec2b20f0>:
resource group "pullsecret-test-8vrs9v6mj4c5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-8vrs9v6mj4c5" to finish deleting: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e9734814-276e-4a55-82bc-7fe55e4a8cd1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e9734814-276e-4a55-82bc-7fe55e4a8cd1",
"name": "e9734814-276e-4a55-82bc-7fe55e4a8cd1",
"status": "Failed",
"startTime": "2026-11-14T13:19:30.200005834Z",
"endTime": "2026-11-14T13:43:33.470145925Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e9734814-276e-4a55-82bc-7fe55e4a8cd1
correlation ID: 3303484d-5e22-4995-a1a3-92df5a0d563f
client request ID: 303745a2-8e9f-4e2f-96c3-31fdb0e1330e
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | alert | 2 | 1.11%2 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-11-18T09:53:00Z Ended: 2026-11-18T10:17:59Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4649472-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.18:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4649472/providers/microsoft.monitor/accounts/hcps-j4649472", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2u5i57o1v2eckvo0n7qq2op477pajdm6-cluster", pod="arohcp-monitor-kube-state-metrics-6785878c5-6c4p4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2u5i57o1v2eckvo0n7qq2op477pajdm6-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-11-18T09:53:00Z Ended: 2026-11-18T10:17:59Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4649472-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.18:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4649472/providers/microsoft.monitor/accounts/hcps-j4649472", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2u5i57o1v2eckvo0n7qq2op477pajdm6-cluster", pod="arohcp-monitor-kube-state-metrics-6785878c5-6c4p4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2u5i57o1v2eckvo0n7qq2op477pajdm6-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-13T13:22:51Z Ended: 2026-11-13T13:28:51Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5993600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5993600/providers/microsoft.monitor/accounts/hcps-j5993600", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster", pod="arohcp-monitor-kube-state-metrics-6785878c5-lvb2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 1.11%2 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 1 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-twvvjrd5h2nj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x32886dfba1c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-twvvjrd5h2nj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j1812736-mgmt-2/ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster router=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003: expected matching summary after test start and within 8m; summaryUID=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003 pass=10e93f8f-93b3-40d9-b741-fa1c25c0d97c at=2026-11-18T08:06:08Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-twvvjrd5h2nj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j1812736-mgmt-2/ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster router=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003: expected matching summary after test start and within 8m; summaryUID=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003 pass=10e93f8f-93b3-40d9-b741-fa1c25c0d97c at=2026-11-18T08:06:08Z",
err: <*verifiers.pollObservationError | 0x32886dfba1a0>{
observation: <*errors.errorString | 0x32886feb6b60>{
s: "dev/centralus/ci01-j1812736-mgmt-2/ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster router=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003: expected matching summary after test start and within 8m; summaryUID=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003 pass=10e93f8f-93b3-40d9-b741-fa1c25c0d97c at=2026-11-18T08:06:08Z",
},
interruption: <*fmt.wrapError | 0x32886dfba180>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1812736-mgmt-2\"; let ns = \"ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster\"; let start = datetime(2026-11-18T07:52:02.909239Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x32886d96c280>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x32886dfba160>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1812736-mgmt-2\"; let ns = \"ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster\"; let start = datetime(2026-11-18T07:52:02.909239Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x32886fc800c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-twvvjrd5h2nj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x32886dfba1c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-twvvjrd5h2nj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j1812736-mgmt-2/ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster router=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003: expected matching summary after test start and within 8m; summaryUID=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003 pass=10e93f8f-93b3-40d9-b741-fa1c25c0d97c at=2026-11-18T08:06:08Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-twvvjrd5h2nj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j1812736-mgmt-2/ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster router=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003: expected matching summary after test start and within 8m; summaryUID=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003 pass=10e93f8f-93b3-40d9-b741-fa1c25c0d97c at=2026-11-18T08:06:08Z",
err: <*verifiers.pollObservationError | 0x32886dfba1a0>{
observation: <*errors.errorString | 0x32886feb6b60>{
s: "dev/centralus/ci01-j1812736-mgmt-2/ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster router=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003: expected matching summary after test start and within 8m; summaryUID=6386c1cc-aa14-4b3e-8250-c1362edd7e8f node=aks-userswft1-11487299-vmss000003 pass=10e93f8f-93b3-40d9-b741-fa1c25c0d97c at=2026-11-18T08:06:08Z",
},
interruption: <*fmt.wrapError | 0x32886dfba180>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1812736-mgmt-2\"; let ns = \"ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster\"; let start = datetime(2026-11-18T07:52:02.909239Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x32886d96c280>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x32886dfba160>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1812736-mgmt-2\"; let ns = \"ocm-arohcpci01-2u5gmssjlr3mq7t0h66n1c2rcc5e5qsf-cluster\"; let start = datetime(2026-11-18T07:52:02.909239Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x32886fc800c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fbjtlbzqs45q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1a4b2543c2e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fbjtlbzqs45q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j4200448-mgmt-1/ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster router=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004: expected matching summary after test start and within 8m; summaryUID=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004 pass=fdae7f46-9f2b-4472-a1e0-2097de8a2fbc at=2026-11-17T06:57:12Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fbjtlbzqs45q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j4200448-mgmt-1/ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster router=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004: expected matching summary after test start and within 8m; summaryUID=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004 pass=fdae7f46-9f2b-4472-a1e0-2097de8a2fbc at=2026-11-17T06:57:12Z",
err: <*verifiers.pollObservationError | 0x1a4b2543c2a0>{
observation: <*errors.errorString | 0x1a4b26becca0>{
s: "dev/centralus/ci01-j4200448-mgmt-1/ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster router=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004: expected matching summary after test start and within 8m; summaryUID=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004 pass=fdae7f46-9f2b-4472-a1e0-2097de8a2fbc at=2026-11-17T06:57:12Z",
},
interruption: <*fmt.wrapError | 0x1a4b2543c280>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4200448-mgmt-1\"; let ns = \"ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster\"; let start = datetime(2026-11-17T06:40:51.2793713Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1a4b243b65f0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1a4b2543c260>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4200448-mgmt-1\"; let ns = \"ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster\"; let start = datetime(2026-11-17T06:40:51.2793713Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1a4b26ec4360>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fbjtlbzqs45q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1a4b2543c2e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fbjtlbzqs45q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j4200448-mgmt-1/ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster router=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004: expected matching summary after test start and within 8m; summaryUID=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004 pass=fdae7f46-9f2b-4472-a1e0-2097de8a2fbc at=2026-11-17T06:57:12Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fbjtlbzqs45q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j4200448-mgmt-1/ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster router=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004: expected matching summary after test start and within 8m; summaryUID=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004 pass=fdae7f46-9f2b-4472-a1e0-2097de8a2fbc at=2026-11-17T06:57:12Z",
err: <*verifiers.pollObservationError | 0x1a4b2543c2a0>{
observation: <*errors.errorString | 0x1a4b26becca0>{
s: "dev/centralus/ci01-j4200448-mgmt-1/ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster router=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004: expected matching summary after test start and within 8m; summaryUID=16e1dd61-73eb-4fd6-bee1-df16bf3e11a3 node=aks-userswft3-40672107-vmss000004 pass=fdae7f46-9f2b-4472-a1e0-2097de8a2fbc at=2026-11-17T06:57:12Z",
},
interruption: <*fmt.wrapError | 0x1a4b2543c280>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4200448-mgmt-1\"; let ns = \"ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster\"; let start = datetime(2026-11-17T06:40:51.2793713Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1a4b243b65f0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1a4b2543c260>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4200448-mgmt-1\"; let ns = \"ocm-arohcpci01-2u4qikkkgjgq156ms5od21i8bkgsk1p0-cluster\"; let start = datetime(2026-11-17T06:40:51.2793713Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1a4b26ec4360>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResour...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 1.11%2 of 180 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x22a3051d8c18>:
resource group "autoscaling-cluster-qrwbnqfsvb2p": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qrwbnqfsvb2p" to finish deleting: GET https://rp.j8171776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b76b8f09-b657-40cb-b784-71d1bde6ff53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b76b8f09-b657-40cb-b784-71d1bde6ff53",
"name": "b76b8f09-b657-40cb-b784-71d1bde6ff53",
"status": "Failed",
"startTime": "2026-11-18T14:37:12.064148646Z",
"endTime": "2026-11-18T15:01:14.347238781Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2u5mcu719hdl7h5sjnreei5ssdb12n30 still exists (deletion dispatched at 2026-11-18T14:37:12Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8171776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b76b8f09-b657-40cb-b784-71d1bde6ff53
correlation ID: 76540f5d-76b0-4660-bd17-0f104b5d1fcf
client request ID: 548c0b6f-f846-4dd4-bc55-b94bc87e0560
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x22a3051d8c18>:
resource group "autoscaling-cluster-qrwbnqfsvb2p": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qrwbnqfsvb2p" to finish deleting: GET https://rp.j8171776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b76b8f09-b657-40cb-b784-71d1bde6ff53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b76b8f09-b657-40cb-b784-71d1bde6ff53",
"name": "b76b8f09-b657-40cb-b784-71d1bde6ff53",
"status": "Failed",
"startTime": "2026-11-18T14:37:12.064148646Z",
"endTime": "2026-11-18T15:01:14.347238781Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2u5mcu719hdl7h5sjnreei5ssdb12n30 still exists (deletion dispatched at 2026-11-18T14:37:12Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8171776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b76b8f09-b657-40cb-b784-71d1bde6ff53
correlation ID: 76540f5d-76b0-4660-bd17-0f104b5d1fcf
client request ID: 548c0b6f-f846-4dd4-bc55-b94bc87e0560
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x6bd18368198>:
resource group "rg-cordon-test-jlf5d8k446r6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-jlf5d8k446r6" to finish deleting: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f6f94e5-d07f-42f2-9909-6f504ee6ae3a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f6f94e5-d07f-42f2-9909-6f504ee6ae3a",
"name": "6f6f94e5-d07f-42f2-9909-6f504ee6ae3a",
"status": "Failed",
"startTime": "2026-11-14T13:18:25.852141473Z",
"endTime": "2026-11-14T13:42:34.171534182Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2u30sp3pqn73llu54jh93u50a9dna30p still exists (deletion dispatched at 2026-11-14T13:18:26Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f6f94e5-d07f-42f2-9909-6f504ee6ae3a
correlation ID: d88a21cf-1681-4909-bef4-9cdcc10ecba9
client request ID: d705fa6b-42d4-4813-be2e-32c8faeb74b0
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x6bd18368198>:
resource group "rg-cordon-test-jlf5d8k446r6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-jlf5d8k446r6" to finish deleting: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f6f94e5-d07f-42f2-9909-6f504ee6ae3a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f6f94e5-d07f-42f2-9909-6f504ee6ae3a",
"name": "6f6f94e5-d07f-42f2-9909-6f504ee6ae3a",
"status": "Failed",
"startTime": "2026-11-14T13:18:25.852141473Z",
"endTime": "2026-11-14T13:42:34.171534182Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2u30sp3pqn73llu54jh93u50a9dna30p still exists (deletion dispatched at 2026-11-14T13:18:26Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6f6f94e5-d07f-42f2-9909-6f504ee6ae3a
correlation ID: d88a21cf-1681-4909-bef4-9cdcc10ecba9
client request ID: d705fa6b-42d4-4813-be2e-32c8faeb74b0
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||
ERROR CODE: ResourceCollectionRequestsThrottled; provider Microsoft.Networkfull failure pattern: ERROR CODE: ResourceCollectionRequestsThrottled; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 1.11%2 of 180 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 2 · Nov 19: 0 | none | |||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:166]: failed to find private KAS internal LB IP in managed resource group "full-priv-kv-586z4fm645nv--managed"
Unexpected error:
<*fmt.wrapError | 0x1a25c2b4bba0>:
failed to list load balancers in "full-priv-kv-586z4fm645nv--managed": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-586z4fm645nv--managed/providers/Microsoft.Network/loadBalancers
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: ResourceCollectionRequestsThrottled
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceCollectionRequestsThrottled",
"message": "Operation 'Microsoft.Network/loadBalancers/read' failed as server encountered too many requests. Please try after '8' seconds. Tracking Id is '01b697f6-7d1e-4b63-96df-925f22cde706'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-586z4fm645nv--managed/providers/Microsoft.Network/loadBalancers
correlation ID: 01b697f6-7d1e-4b63-96df-925f22cde706
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:166]: failed to find private KAS internal LB IP in managed resource group "full-priv-kv-586z4fm645nv--managed"
Unexpected error:
<*fmt.wrapError | 0x1a25c2b4bba0>:
failed to list load balancers in "full-priv-kv-586z4fm645nv--managed": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-586z4fm645nv--managed/providers/Microsoft.Network/loadBalancers
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: ResourceCollectionRequestsThrottled
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceCollectionRequestsThrottled",
"message": "Operation 'Microsoft.Network/loadBalancers/read' failed as server encountered too many requests. Please try after '8' seconds. Tracking Id is '01b697f6-7d1e-4b63-96df-925f22cde706'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-586z4fm645nv--managed/providers/Microsoft.Network/loadBalancers
correlation ID: 01b697f6-7d1e-4b63-96df-925f22cde706
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x30a224028138>:
resource group "ingress-cert-le-gndngkvmggml": failed to cleanup resource group: failed to detach NSGs from subnets in resource group ingress-cert-le-gndngkvmggml: failed listing vnets in resource group ingress-cert-le-gndngkvmggml: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gndngkvmggml/providers/Microsoft.Network/virtualNetworks
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: ResourceCollectionRequestsThrottled
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceCollectionRequestsThrottled",
"message": "Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '5' seconds. Tracking Id is 'bb4e3d40-f65e-4094-aac8-64adbf0e1555'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gndngkvmggml/providers/Microsoft.Network/virtualNetworks
correlation ID: bb4e3d40-f65e-4094-aac8-64adbf0e1555
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x30a224028138>:
resource group "ingress-cert-le-gndngkvmggml": failed to cleanup resource group: failed to detach NSGs from subnets in resource group ingress-cert-le-gndngkvmggml: failed listing vnets in resource group ingress-cert-le-gndngkvmggml: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gndngkvmggml/providers/Microsoft.Network/virtualNetworks
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: ResourceCollectionRequestsThrottled
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceCollectionRequestsThrottled",
"message": "Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '5' seconds. Tracking Id is 'bb4e3d40-f65e-4094-aac8-64adbf0e1555'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gndngkvmggml/providers/Microsoft.Network/virtualNetworks
correlation ID: bb4e3d40-f65e-4094-aac8-64adbf0e1555
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 4.76%1 of 21 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | DEV, PROD | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x17a04ae8aee0>:
failed to create HCP cluster np-dg-minor-89bskt: failed waiting for cluster="np-dg-minor-89bskt" in resourcegroup="rg-np-dg-minor-89bskt-sv5nkfqrjg8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4",
"name": "7966bc18-a5b0-4651-b41c-fc426eb273c4",
"status": "Failed",
"startTime": "2026-11-14T20:41:04.80974955Z",
"endTime": "2026-11-14T21:10:07.827174067Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
correlation ID: 9c07c544-07d2-4418-852b-26b1b9c7f5ee
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x17a04ae8aee0>:
failed to create HCP cluster np-dg-minor-89bskt: failed waiting for cluster="np-dg-minor-89bskt" in resourcegroup="rg-np-dg-minor-89bskt-sv5nkfqrjg8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4",
"name": "7966bc18-a5b0-4651-b41c-fc426eb273c4",
"status": "Failed",
"startTime": "2026-11-14T20:41:04.80974955Z",
"endTime": "2026-11-14T21:10:07.827174067Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
correlation ID: 9c07c544-07d2-4418-852b-26b1b9c7f5ee
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining res...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.76%1 of 21 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x1a1012fe330>:
resource group "feature-aggregation-b6fvjljtlcvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-b6fvjljtlcvs" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c",
"name": "e2b882db-48de-4ff2-9f50-4222b9955e6c",
"status": "Failed",
"startTime": "2026-11-16T02:36:07.442208668Z",
"endTime": "2026-11-16T03:00:10.537191563Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
correlation ID: 9b1360be-096b-4387-8316-508cc19a90f1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x1a1012fe330>:
resource group "feature-aggregation-b6fvjljtlcvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-b6fvjljtlcvs" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c",
"name": "e2b882db-48de-4ff2-9f50-4222b9955e6c",
"status": "Failed",
"startTime": "2026-11-16T02:36:07.442208668Z",
"endTime": "2026-11-16T03:00:10.537191563Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
correlation ID: 9b1360be-096b-4387-8316-508cc19a90f1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.76%1 of 21 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:223]: failed to revoke admin credentials for cluster "admin-cred-lifecycle-vgkc2m"
Unexpected error:
<*fmt.wrapErrors | 0x41bbce44ea0>:
failed waiting for hcpCluster="admin-cred-lifecycle-vgkc2m" in resourcegroup="admin-credential-lifecycle-test-bgcs6sp9q9hf" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-vgkc2m\" in resourcegroup=\"admin-credential-lifecycle-test-bgcs6sp9q9hf\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x41bbbe36b00>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:223]: failed to revoke admin credentials for cluster "admin-cred-lifecycle-vgkc2m"
Unexpected error:
<*fmt.wrapErrors | 0x41bbce44ea0>:
failed waiting for hcpCluster="admin-cred-lifecycle-vgkc2m" in resourcegroup="admin-credential-lifecycle-test-bgcs6sp9q9hf" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-vgkc2m\" in resourcegroup=\"admin-credential-lifecycle-test-bgcs6sp9q9hf\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x41bbbe36b00>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
Unauthorizedfull failure pattern: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.76%1 of 21 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:236]: Timed out after 300.001s.
Expected success, but got an error:
<*errors.StatusError | 0x38a7b05b720>:
Unauthorized
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:236]: Timed out after 300.001s.
Expected success, but got an error:
<*errors.StatusError | 0x38a7b05b720>:
Unauthorized
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod <namespace>/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the r...full failure pattern: VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod <namespace>/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the request from succeeding (get pod <pod>)Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.76%1 of 21 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-20-tc28nz"
Unexpected error:
<*errors.joinError | 0x227e60b84150>:
VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the request from succeeding (get pods e2e-csi-pod)
{
errs: [
<*fmt.wrapError | 0x227e601b6be0>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*fmt.wrapError | 0x227e6113c8c0>{
msg: "failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*errors.StatusError | 0x227e60a46dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
Reason: "InternalError",
Details: {
Name: "e2e-csi-pod",
Group: "",
Kind: "pods",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "unknown",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-20-tc28nz"
Unexpected error:
<*errors.joinError | 0x227e60b84150>:
VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the request from succeeding (get pods e2e-csi-pod)
{
errs: [
<*fmt.wrapError | 0x227e601b6be0>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*fmt.wrapError | 0x227e6113c8c0>{
msg: "failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*errors.StatusError | 0x227e60a46dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
Reason: "InternalError",
Details: {
Name: "e2e-csi-pod",
Group: "",
Kind: "pods",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "unknown",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp...full failure pattern: VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.76%1 of 21 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-4wrz4s"
Unexpected error:
<*fmt.wrapError | 0x244b4deba960>:
VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-4fxqjqb"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-4fxqjqb\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase=\"Running\" reason=\"\"",
err: <*verifiers.pollObservationError | 0x244b4deea6c0>{
observation: <*errors.errorString | 0x244b4fb06d40>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net",
},
interruption: <*fmt.wrapError | 0x244b4deea660>{
msg: "console login redirect: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-4wrz4s"
Unexpected error:
<*fmt.wrapError | 0x244b4deba960>:
VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-4fxqjqb"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-4fxqjqb\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase=\"Running\" reason=\"\"",
err: <*verifiers.pollObservationError | 0x244b4deea6c0>{
observation: <*errors.errorString | 0x244b4fb06d40>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net",
},
interruption: <*fmt.wrapError | 0x244b4deea660>{
msg: "console login redirect: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
Environment: STG
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 25.00%2 of 8 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | PROD | |||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:138]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
<*errors.joinError | 0x10cb1baa2f30>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x10cb1bdb4aa0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x10cb1bdb4a80>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x10cb1c483040>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x10cb1bc13be0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x10cb1bc13bc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x10cb1b7c5540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:138]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
<*errors.joinError | 0x10cb1baa2f30>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x10cb1bdb4aa0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x10cb1bdb4a80>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x10cb1c483040>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x10cb1bc13be0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x10cb1bc13bc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x10cb1b7c5540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x2b922c8b8090>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2b922c896040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2b922be8e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2b922c8960a0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896080>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2b922be8efa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x2b922c8b8090>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2b922c896040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2b922be8e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2b922c8960a0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896080>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2b922be8efa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 12.50%1 of 8 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | DEV, PROD | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:149]: failed to verify cluster "cp-ystream-upgrade-5-0-jgz7z5" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x3e96a1c86db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e96a1d21c80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e96a1d21c60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e96a28e4820>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:149]: failed to verify cluster "cp-ystream-upgrade-5-0-jgz7z5" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x3e96a1c86db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e96a1d21c80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e96a1d21c60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e96a28e4820>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Conflict; detail message The request failed due to conflict with a concurrent request. To resolve it, pleasefull failure pattern: ERROR CODE: DeploymentFailed; detail code Conflict; detail message The request failed due to conflict with a concurrent request. To resolve it, please Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 12.50%1 of 8 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:165]: failed to deploy authorized CIDR VM
Unexpected error:
<*fmt.wrapError | 0x2c2acec007e0>:
failed waiting for deployment "test-vm" in resourcegroup="feature-aggregation-bz4zrh6ch5d7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-bz4zrh6ch5d7/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584066501361417598
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b5242f08-a22e-4a83-a2ed-a730a9204454",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"Conflict\",\r\n \"message\": \"The request failed due to conflict with a concurrent request. To resolve it, please refer to https://aka.ms/activitylog to get more details on the conflicting requests.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-bz4zrh6ch5d7/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584066501361417598
correlation ID: 9229caa2-fc9a-41de-8c36-051176cec6a0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:165]: failed to deploy authorized CIDR VM
Unexpected error:
<*fmt.wrapError | 0x2c2acec007e0>:
failed waiting for deployment "test-vm" in resourcegroup="feature-aggregation-bz4zrh6ch5d7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-bz4zrh6ch5d7/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584066501361417598
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b5242f08-a22e-4a83-a2ed-a730a9204454",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"Conflict\",\r\n \"message\": \"The request failed due to conflict with a concurrent request. To resolve it, please refer to https://aka.ms/activitylog to get more details on the conflicting requests.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-bz4zrh6ch5d7/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584066501361417598
correlation ID: 9229caa2-fc9a-41de-8c36-051176cec6a0
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 6 | 22.22%6 of 27 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 3 · Nov 18: 2 · Nov 19: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x1f1971f8a680>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4crcvw4nmfpc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/b1fd32c7-69db-439b-a7a2-21e7941c7086
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/b1fd32c7-69db-439b-a7a2-21e7941c7086",
"name": "b1fd32c7-69db-439b-a7a2-21e7941c7086",
"status": "Failed",
"startTime": "2026-11-18T05:01:04.424170895Z",
"endTime": "2026-11-18T05:20:06.274082307Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-delete-main-jpt7c-8x62f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-delete-main-jpt7c-rchss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-delete-main-jpt7c-8x62f: Creating: virtualmachine creating or updating\nMachine cluster-np-delete-main-jpt7c-rchss: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/b1fd32c7-69db-439b-a7a2-21e7941c7086
correlation ID: 16184f17-6e98-488a-93b4-483b69050df8
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x1f1971f8a680>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4crcvw4nmfpc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/b1fd32c7-69db-439b-a7a2-21e7941c7086
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/b1fd32c7-69db-439b-a7a2-21e7941c7086",
"name": "b1fd32c7-69db-439b-a7a2-21e7941c7086",
"status": "Failed",
"startTime": "2026-11-18T05:01:04.424170895Z",
"endTime": "2026-11-18T05:20:06.274082307Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-delete-main-jpt7c-8x62f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-delete-main-jpt7c-rchss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-delete-main-jpt7c-8x62f: Creating: virtualmachine creating or updating\nMachine cluster-np-delete-main-jpt7c-rchss: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/b1fd32c7-69db-439b-a7a2-21e7941c7086
correlation ID: 16184f17-6e98-488a-93b4-483b69050df8
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xf3885770020>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-dgzzffndnt44" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/fd78270a-bdd2-4795-9d98-1240a5cee081
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/fd78270a-bdd2-4795-9d98-1240a5cee081",
"name": "fd78270a-bdd2-4795-9d98-1240a5cee081",
"status": "Failed",
"startTime": "2026-11-18T01:44:18.327657265Z",
"endTime": "2026-11-18T02:03:19.265192766Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-22wq7-2mcfr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-22wq7-gv5mg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-22wq7-2mcfr: Failed: virtualmachine failed to create or update. err: failed to create or update resource ingress-cert-le-dgzzffndnt44--managed/cluster-np-1-22wq7-2mcfr (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-dgzzffndnt44--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-22wq7-2mcfr\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum) New Limit Requi... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/fd78270a-bdd2-4795-9d98-1240a5cee081
correlation ID: d639fa73-81c2-4451-946c-cb49bf42a66b
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xf3885770020>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-dgzzffndnt44" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/fd78270a-bdd2-4795-9d98-1240a5cee081
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/fd78270a-bdd2-4795-9d98-1240a5cee081",
"name": "fd78270a-bdd2-4795-9d98-1240a5cee081",
"status": "Failed",
"startTime": "2026-11-18T01:44:18.327657265Z",
"endTime": "2026-11-18T02:03:19.265192766Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-22wq7-2mcfr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-22wq7-gv5mg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-22wq7-2mcfr: Failed: virtualmachine failed to create or update. err: failed to create or update resource ingress-cert-le-dgzzffndnt44--managed/cluster-np-1-22wq7-2mcfr (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-dgzzffndnt44--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-22wq7-2mcfr\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum) New Limit Requi... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/fd78270a-bdd2-4795-9d98-1240a5cee081
correlation ID: d639fa73-81c2-4451-946c-cb49bf42a66b
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2f6ef11e3660>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jt8q49cnjl2q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/42b7c0c1-51d9-4524-8635-8e849cf0359a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/42b7c0c1-51d9-4524-8635-8e849cf0359a",
"name": "42b7c0c1-51d9-4524-8635-8e849cf0359a",
"status": "Failed",
"startTime": "2026-11-17T19:48:10.99784096Z",
"endTime": "2026-11-17T20:07:14.675639432Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-deletable-2b9bq-6zv2h: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-deletable-2b9bq-9zqnz: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-deletable-2b9bq-6zv2h: Creating: virtualmachine creating or updating\nMachine cluster-np-deletable-2b9bq-9zqnz: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/42b7c0c1-51d9-4524-8635-8e849cf0359a
correlation ID: 05a25853-57b6-41b4-b8da-b1b39fd045ba
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2f6ef11e3660>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jt8q49cnjl2q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/42b7c0c1-51d9-4524-8635-8e849cf0359a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/42b7c0c1-51d9-4524-8635-8e849cf0359a",
"name": "42b7c0c1-51d9-4524-8635-8e849cf0359a",
"status": "Failed",
"startTime": "2026-11-17T19:48:10.99784096Z",
"endTime": "2026-11-17T20:07:14.675639432Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-deletable-2b9bq-6zv2h: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-deletable-2b9bq-9zqnz: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-deletable-2b9bq-6zv2h: Creating: virtualmachine creating or updating\nMachine cluster-np-deletable-2b9bq-9zqnz: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/42b7c0c1-51d9-4524-8635-8e849cf0359a
correlation ID: 05a25853-57b6-41b4-b8da-b1b39fd045ba
client request ID: unavailableContributing tests (4)
Affected runs (6)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 5 | 18.52%5 of 27 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 12: 0 · Nov 13: 6 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 24 · Nov 18: 12 · Nov 19: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-njzj67" for cluster "cluster-candidate-4-21-njzj67"
Unexpected error:
<*fmt.wrapError | 0x3fb97e354000>:
failed to create NodePool np-njzj67: failed waiting for nodepool="np-njzj67" for cluster "cluster-candidate-4-21-njzj67" in resourcegroup="rg-candidate-4-21-2c48lxsq2lbd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/21456a82-7a6a-4790-8b87-a30a2dbf10a7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/21456a82-7a6a-4790-8b87-a30a2dbf10a7",
"name": "21456a82-7a6a-4790-8b87-a30a2dbf10a7",
"status": "Failed",
"startTime": "2026-11-18T04:34:52.136505419Z",
"endTime": "2026-11-18T04:53:53.00644935Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-njzj67-b4gj2-c6mwr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-njzj67-b4gj2-q9kff: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-njzj67-b4gj2-c6mwr: Creating: virtualmachine creating or updating\nMachine cluster-np-njzj67-b4gj2-q9kff: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/21456a82-7a6a-4790-8b87-a30a2dbf10a7
correlation ID: 2242ad97-b5e9-44da-b568-7ea01bfdb058
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-njzj67" for cluster "cluster-candidate-4-21-njzj67"
Unexpected error:
<*fmt.wrapError | 0x3fb97e354000>:
failed to create NodePool np-njzj67: failed waiting for nodepool="np-njzj67" for cluster "cluster-candidate-4-21-njzj67" in resourcegroup="rg-candidate-4-21-2c48lxsq2lbd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/21456a82-7a6a-4790-8b87-a30a2dbf10a7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/21456a82-7a6a-4790-8b87-a30a2dbf10a7",
"name": "21456a82-7a6a-4790-8b87-a30a2dbf10a7",
"status": "Failed",
"startTime": "2026-11-18T04:34:52.136505419Z",
"endTime": "2026-11-18T04:53:53.00644935Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-njzj67-b4gj2-c6mwr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-njzj67-b4gj2-q9kff: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-njzj67-b4gj2-c6mwr: Creating: virtualmachine creating or updating\nMachine cluster-np-njzj67-b4gj2-q9kff: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/21456a82-7a6a-4790-8b87-a30a2dbf10a7
correlation ID: 2242ad97-b5e9-44da-b568-7ea01bfdb058
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-9fnpv5" for cluster "cluster-candidate-4-20-9fnpv5"
Unexpected error:
<*fmt.wrapError | 0x11172788a420>:
failed to create NodePool np-9fnpv5: failed waiting for nodepool="np-9fnpv5" for cluster "cluster-candidate-4-20-9fnpv5" in resourcegroup="rg-candidate-4-20-mp244w8v7fmn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d80d78dd-ffd8-42bb-8f10-df0198d656eb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d80d78dd-ffd8-42bb-8f10-df0198d656eb",
"name": "d80d78dd-ffd8-42bb-8f10-df0198d656eb",
"status": "Failed",
"startTime": "2026-11-18T04:31:45.843384317Z",
"endTime": "2026-11-18T04:50:52.881106436Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-9fnpv5-4hmqs-8b7r7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-9fnpv5-4hmqs-wnfj8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-9fnpv5-4hmqs-8b7r7: Creating: virtualmachine creating or updating\nMachine cluster-np-9fnpv5-4hmqs-wnfj8: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d80d78dd-ffd8-42bb-8f10-df0198d656eb
correlation ID: fc11e261-832c-4b02-ac6d-f7d4c75eb351
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-9fnpv5" for cluster "cluster-candidate-4-20-9fnpv5"
Unexpected error:
<*fmt.wrapError | 0x11172788a420>:
failed to create NodePool np-9fnpv5: failed waiting for nodepool="np-9fnpv5" for cluster "cluster-candidate-4-20-9fnpv5" in resourcegroup="rg-candidate-4-20-mp244w8v7fmn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d80d78dd-ffd8-42bb-8f10-df0198d656eb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d80d78dd-ffd8-42bb-8f10-df0198d656eb",
"name": "d80d78dd-ffd8-42bb-8f10-df0198d656eb",
"status": "Failed",
"startTime": "2026-11-18T04:31:45.843384317Z",
"endTime": "2026-11-18T04:50:52.881106436Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-9fnpv5-4hmqs-8b7r7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-9fnpv5-4hmqs-wnfj8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-9fnpv5-4hmqs-8b7r7: Creating: virtualmachine creating or updating\nMachine cluster-np-9fnpv5-4hmqs-wnfj8: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d80d78dd-ffd8-42bb-8f10-df0198d656eb
correlation ID: fc11e261-832c-4b02-ac6d-f7d4c75eb351
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-946qjg" for cluster "cluster-candidate-5-0-946qjg"
Unexpected error:
<*fmt.wrapError | 0xc40b8381e0>:
failed to create NodePool np-946qjg: failed waiting for nodepool="np-946qjg" for cluster "cluster-candidate-5-0-946qjg" in resourcegroup="rg-candidate-5-0-mtdmdkp4pqw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/701bb11f-bcc0-4303-94ec-d55094888aff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/701bb11f-bcc0-4303-94ec-d55094888aff",
"name": "701bb11f-bcc0-4303-94ec-d55094888aff",
"status": "Failed",
"startTime": "2026-11-18T04:35:19.386020316Z",
"endTime": "2026-11-18T04:54:22.981200829Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-946qjg-mhhwl-4gmk9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-946qjg-mhhwl-dmldw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-946qjg-mhhwl-4gmk9: Creating: virtualmachine creating or updating\nMachine cluster-np-946qjg-mhhwl-dmldw: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/701bb11f-bcc0-4303-94ec-d55094888aff
correlation ID: ce482e4f-34b2-43d1-8189-a33229d16177
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-946qjg" for cluster "cluster-candidate-5-0-946qjg"
Unexpected error:
<*fmt.wrapError | 0xc40b8381e0>:
failed to create NodePool np-946qjg: failed waiting for nodepool="np-946qjg" for cluster "cluster-candidate-5-0-946qjg" in resourcegroup="rg-candidate-5-0-mtdmdkp4pqw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/701bb11f-bcc0-4303-94ec-d55094888aff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/701bb11f-bcc0-4303-94ec-d55094888aff",
"name": "701bb11f-bcc0-4303-94ec-d55094888aff",
"status": "Failed",
"startTime": "2026-11-18T04:35:19.386020316Z",
"endTime": "2026-11-18T04:54:22.981200829Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-946qjg-mhhwl-4gmk9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-946qjg-mhhwl-dmldw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-946qjg-mhhwl-4gmk9: Creating: virtualmachine creating or updating\nMachine cluster-np-946qjg-mhhwl-dmldw: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/701bb11f-bcc0-4303-94ec-d55094888aff
correlation ID: ce482e4f-34b2-43d1-8189-a33229d16177
client request ID: unavailable
occurredContributing tests (14)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 4 | 14.81%4 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 2 · Nov 18: 1 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x2b3ab11d40c0>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-d5kggtpdjx2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d4958751-de24-4923-91b2-57a057de365c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d4958751-de24-4923-91b2-57a057de365c",
"name": "d4958751-de24-4923-91b2-57a057de365c",
"status": "Failed",
"startTime": "2026-11-18T04:11:18.085637178Z",
"endTime": "2026-11-18T04:30:22.753216556Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-agg-np-a-cq2p5-klnhn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-agg-np-a-cq2p5-msxtx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-agg-np-a-cq2p5-klnhn: Creating: virtualmachine creating or updating\nMachine cluster-agg-np-a-cq2p5-msxtx: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d4958751-de24-4923-91b2-57a057de365c
correlation ID: 196d55c3-2462-49f8-a5b7-0b3b17c1859f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x2b3ab11d40c0>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-d5kggtpdjx2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d4958751-de24-4923-91b2-57a057de365c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d4958751-de24-4923-91b2-57a057de365c",
"name": "d4958751-de24-4923-91b2-57a057de365c",
"status": "Failed",
"startTime": "2026-11-18T04:11:18.085637178Z",
"endTime": "2026-11-18T04:30:22.753216556Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-agg-np-a-cq2p5-klnhn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-agg-np-a-cq2p5-msxtx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-agg-np-a-cq2p5-klnhn: Creating: virtualmachine creating or updating\nMachine cluster-agg-np-a-cq2p5-msxtx: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/d4958751-de24-4923-91b2-57a057de365c
correlation ID: 196d55c3-2462-49f8-a5b7-0b3b17c1859f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xcb68f014030>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rw99jqnqbxp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/80fb5b5d-def6-4d6c-8758-727b9e8655e1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/80fb5b5d-def6-4d6c-8758-727b9e8655e1",
"name": "80fb5b5d-def6-4d6c-8758-727b9e8655e1",
"status": "Failed",
"startTime": "2026-11-17T18:57:18.011593246Z",
"endTime": "2026-11-17T19:16:19.382207886Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-agg-np-a-kbjzj-dcbpb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-agg-np-a-kbjzj-tvk8v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-agg-np-a-kbjzj-dcbpb: Creating: virtualmachine creating or updating\nMachine cluster-agg-np-a-kbjzj-tvk8v: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/80fb5b5d-def6-4d6c-8758-727b9e8655e1
correlation ID: 4411fa03-01cc-4708-b652-cf6ee3effccc
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xcb68f014030>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rw99jqnqbxp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/80fb5b5d-def6-4d6c-8758-727b9e8655e1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/80fb5b5d-def6-4d6c-8758-727b9e8655e1",
"name": "80fb5b5d-def6-4d6c-8758-727b9e8655e1",
"status": "Failed",
"startTime": "2026-11-17T18:57:18.011593246Z",
"endTime": "2026-11-17T19:16:19.382207886Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-agg-np-a-kbjzj-dcbpb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-agg-np-a-kbjzj-tvk8v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-agg-np-a-kbjzj-dcbpb: Creating: virtualmachine creating or updating\nMachine cluster-agg-np-a-kbjzj-tvk8v: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/80fb5b5d-def6-4d6c-8758-727b9e8655e1
correlation ID: 4411fa03-01cc-4708-b652-cf6ee3effccc
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x1649f3ed4018>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-jk5x8bjqf4z7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/f1cb486e-23a1-4c49-a2cb-c70c34e00dfc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/f1cb486e-23a1-4c49-a2cb-c70c34e00dfc",
"name": "f1cb486e-23a1-4c49-a2cb-c70c34e00dfc",
"status": "Failed",
"startTime": "2026-11-17T10:00:15.985394864Z",
"endTime": "2026-11-17T10:19:21.910929337Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-agg-np-a-74sj7-rnw7k: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-agg-np-a-74sj7-wrm7h: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-agg-np-a-74sj7-rnw7k: Creating: virtualmachine creating or updating\nMachine cluster-agg-np-a-74sj7-wrm7h: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/f1cb486e-23a1-4c49-a2cb-c70c34e00dfc
correlation ID: d4a3a1a2-0971-477e-843f-8e278240f58e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x1649f3ed4018>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-jk5x8bjqf4z7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/f1cb486e-23a1-4c49-a2cb-c70c34e00dfc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/f1cb486e-23a1-4c49-a2cb-c70c34e00dfc",
"name": "f1cb486e-23a1-4c49-a2cb-c70c34e00dfc",
"status": "Failed",
"startTime": "2026-11-17T10:00:15.985394864Z",
"endTime": "2026-11-17T10:19:21.910929337Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-agg-np-a-74sj7-rnw7k: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-agg-np-a-74sj7-wrm7h: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-agg-np-a-74sj7-rnw7k: Creating: virtualmachine creating or updating\nMachine cluster-agg-np-a-74sj7-wrm7h: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/f1cb486e-23a1-4c49-a2cb-c70c34e00dfc
correlation ID: d4a3a1a2-0971-477e-843f-8e278240f58e
client request ID: unavailable
occurredContributing tests (2)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 3 | 11.11%3 of 27 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsNov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 1 · Nov 16: 1 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:255]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x398e0fc00660>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x398e0fbc1140>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x398e0fbc0000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x398e0f6ca960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x398e100a8b80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x398e100a8b60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x398e0f4ba500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:255]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x398e0fc00660>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x398e0fbc1140>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x398e0fbc0000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x398e0f6ca960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x398e100a8b80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x398e100a8b60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x398e0f4ba500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x1391d91aa9d8>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1391d921f1a0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1391d921f180>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1391d8f254a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1391d9035c00>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1391d9035be0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1391d90379a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x1391d91aa9d8>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1391d921f1a0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1391d921f180>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1391d8f254a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1391d9035c00>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1391d9035be0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1391d90379a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:201]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x1883079cea80>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x188307f50040>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x188307f50020>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x18830755e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1883079de2c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1883079de2a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1883073ec960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:201]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x1883079cea80>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x188307f50040>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x188307f50020>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x18830755e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1883079de2c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1883079de2a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1883073ec960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 11.11%3 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 52 · Nov 18: 106 · Nov 19: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1fc44c184700>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-fnx95dswj9z6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/46170d10-c3be-4d63-8959-67d70b6bf37c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/46170d10-c3be-4d63-8959-67d70b6bf37c",
"name": "46170d10-c3be-4d63-8959-67d70b6bf37c",
"status": "Failed",
"startTime": "2026-11-18T13:03:02.135337964Z",
"endTime": "2026-11-18T13:32:10.199828224Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/46170d10-c3be-4d63-8959-67d70b6bf37c
correlation ID: b4a2267d-f828-4c8d-840e-5c285df9eaf7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1fc44c184700>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-fnx95dswj9z6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/46170d10-c3be-4d63-8959-67d70b6bf37c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/46170d10-c3be-4d63-8959-67d70b6bf37c",
"name": "46170d10-c3be-4d63-8959-67d70b6bf37c",
"status": "Failed",
"startTime": "2026-11-18T13:03:02.135337964Z",
"endTime": "2026-11-18T13:32:10.199828224Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/46170d10-c3be-4d63-8959-67d70b6bf37c
correlation ID: b4a2267d-f828-4c8d-840e-5c285df9eaf7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-jjkvgk4s4pht/cluster-candidate-4-21-nznbl8 should provision
Unexpected error:
<*fmt.wrapError | 0x1afa73133340>:
failed to create HCP cluster cluster-candidate-4-21-nznbl8: failed waiting for cluster="cluster-candidate-4-21-nznbl8" in resourcegroup="rg-candidate-4-21-jjkvgk4s4pht" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/ea0917f0-03cc-4e2c-854c-df4c4d577c82
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/ea0917f0-03cc-4e2c-854c-df4c4d577c82",
"name": "ea0917f0-03cc-4e2c-854c-df4c4d577c82",
"status": "Failed",
"startTime": "2026-11-18T12:22:53.929578627Z",
"endTime": "2026-11-18T12:51:53.652842352Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/ea0917f0-03cc-4e2c-854c-df4c4d577c82
correlation ID: dcd688ac-0a80-4437-8725-31827b1b7a1f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-jjkvgk4s4pht/cluster-candidate-4-21-nznbl8 should provision
Unexpected error:
<*fmt.wrapError | 0x1afa73133340>:
failed to create HCP cluster cluster-candidate-4-21-nznbl8: failed waiting for cluster="cluster-candidate-4-21-nznbl8" in resourcegroup="rg-candidate-4-21-jjkvgk4s4pht" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/ea0917f0-03cc-4e2c-854c-df4c4d577c82
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/ea0917f0-03cc-4e2c-854c-df4c4d577c82",
"name": "ea0917f0-03cc-4e2c-854c-df4c4d577c82",
"status": "Failed",
"startTime": "2026-11-18T12:22:53.929578627Z",
"endTime": "2026-11-18T12:51:53.652842352Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/ea0917f0-03cc-4e2c-854c-df4c4d577c82
correlation ID: dcd688ac-0a80-4437-8725-31827b1b7a1f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3b9edc39c980>:
failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-rvh6dnsfngr5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/d55dd596-2ed0-4fb0-9d55-b61cdb92ffff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/d55dd596-2ed0-4fb0-9d55-b61cdb92ffff",
"name": "d55dd596-2ed0-4fb0-9d55-b61cdb92ffff",
"status": "Failed",
"startTime": "2026-11-18T12:58:37.131360889Z",
"endTime": "2026-11-18T13:27:40.200865582Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/d55dd596-2ed0-4fb0-9d55-b61cdb92ffff
correlation ID: 2355d43c-87b4-4868-ab0d-f1452fbd3aac
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3b9edc39c980>:
failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-rvh6dnsfngr5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/d55dd596-2ed0-4fb0-9d55-b61cdb92ffff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/d55dd596-2ed0-4fb0-9d55-b61cdb92ffff",
"name": "d55dd596-2ed0-4fb0-9d55-b61cdb92ffff",
"status": "Failed",
"startTime": "2026-11-18T12:58:37.131360889Z",
"endTime": "2026-11-18T13:27:40.200865582Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/d55dd596-2ed0-4fb0-9d55-b61cdb92ffff
correlation ID: 2355d43c-87b4-4868-ab0d-f1452fbd3aac
client request ID: unavailable
occurredContributing tests (67)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 11.11%3 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 9 · Nov 18: 20 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3267e92862c0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-prqrgw52mfdj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/33d57277-2be3-4cb3-94df-dabaa12904fd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/33d57277-2be3-4cb3-94df-dabaa12904fd",
"name": "33d57277-2be3-4cb3-94df-dabaa12904fd",
"status": "Failed",
"startTime": "2026-11-18T11:46:02.198458505Z",
"endTime": "2026-11-18T12:15:10.614354913Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/33d57277-2be3-4cb3-94df-dabaa12904fd
correlation ID: 9242865a-2bd7-4fe3-91f3-6ec3138ba663
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3267e92862c0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-prqrgw52mfdj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/33d57277-2be3-4cb3-94df-dabaa12904fd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/33d57277-2be3-4cb3-94df-dabaa12904fd",
"name": "33d57277-2be3-4cb3-94df-dabaa12904fd",
"status": "Failed",
"startTime": "2026-11-18T11:46:02.198458505Z",
"endTime": "2026-11-18T12:15:10.614354913Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/33d57277-2be3-4cb3-94df-dabaa12904fd
correlation ID: 9242865a-2bd7-4fe3-91f3-6ec3138ba663
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xc2075e0d9c0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-n7hplpdw69m8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/692cfdde-9334-4df5-a8a1-61428b91e256
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/692cfdde-9334-4df5-a8a1-61428b91e256",
"name": "692cfdde-9334-4df5-a8a1-61428b91e256",
"status": "Failed",
"startTime": "2026-11-18T11:46:03.103498356Z",
"endTime": "2026-11-18T12:15:10.391090043Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/692cfdde-9334-4df5-a8a1-61428b91e256
correlation ID: 66468d27-0feb-42e8-9941-ab03c1d13c72
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xc2075e0d9c0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-n7hplpdw69m8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/692cfdde-9334-4df5-a8a1-61428b91e256
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/692cfdde-9334-4df5-a8a1-61428b91e256",
"name": "692cfdde-9334-4df5-a8a1-61428b91e256",
"status": "Failed",
"startTime": "2026-11-18T11:46:03.103498356Z",
"endTime": "2026-11-18T12:15:10.391090043Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/692cfdde-9334-4df5-a8a1-61428b91e256
correlation ID: 66468d27-0feb-42e8-9941-ab03c1d13c72
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3c4a09250300>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-xgkmvjplbr5m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/9808a00c-e15a-434c-a22f-be87aab65c7a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/9808a00c-e15a-434c-a22f-be87aab65c7a",
"name": "9808a00c-e15a-434c-a22f-be87aab65c7a",
"status": "Failed",
"startTime": "2026-11-18T11:46:20.19174644Z",
"endTime": "2026-11-18T12:15:20.384166078Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/9808a00c-e15a-434c-a22f-be87aab65c7a
correlation ID: 4a017b43-d66c-4d73-a913-4f93a09fc293
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3c4a09250300>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-xgkmvjplbr5m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/9808a00c-e15a-434c-a22f-be87aab65c7a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/9808a00c-e15a-434c-a22f-be87aab65c7a",
"name": "9808a00c-e15a-434c-a22f-be87aab65c7a",
"status": "Failed",
"startTime": "2026-11-18T11:46:20.19174644Z",
"endTime": "2026-11-18T12:15:20.384166078Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is validating; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/9808a00c-e15a-434c-a22f-be87aab65c7a
correlation ID: 4a017b43-d66c-4d73-a913-4f93a09fc293
client request ID: unavailable
occurredContributing tests (21)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 3 | 11.11%3 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 2 · Nov 19: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 3 | 11.11%3 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 2 · Nov 19: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xf718ba64330>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-f7k4m5b9sz7f" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-f7k4m5b9sz7f, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-f7k4m5b9sz7f\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-f7k4m5b9sz7f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xf718c57b8d0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-f7k4m5b9sz7f",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xf718ba64330>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-f7k4m5b9sz7f" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-f7k4m5b9sz7f, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-f7k4m5b9sz7f\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-f7k4m5b9sz7f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xf718c57b8d0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-f7k4m5b9sz7f",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x1d979d42480>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zzvzmlk68szv" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zzvzmlk68szv, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zzvzmlk68szv\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zzvzmlk68szv, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1d979f6e430>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zzvzmlk68szv",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x1d979d42480>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zzvzmlk68szv" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zzvzmlk68szv, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zzvzmlk68szv\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zzvzmlk68szv, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1d979f6e430>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zzvzmlk68szv",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x8ea9736c360>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-2trr5v6jpvdx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2trr5v6jpvdx, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-2trr5v6jpvdx\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2trr5v6jpvdx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x8ea971f8fa0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2trr5v6jpvdx",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x8ea9736c360>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-2trr5v6jpvdx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2trr5v6jpvdx, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-2trr5v6jpvdx\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2trr5v6jpvdx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x8ea971f8fa0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2trr5v6jpvdx",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 7.41%2 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 2 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x38b8c17a6b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8h7cslrzs4d4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"name": "cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"status": "Failed",
"startTime": "2026-11-13T07:22:34.538735812Z",
"endTime": "2026-11-13T07:41:41.602384099Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-lv9kw-7q9mn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-lv9kw-7q9mn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
correlation ID: 9af047e6-a985-412b-a5f1-08a259c865d2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x38b8c17a6b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8h7cslrzs4d4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"name": "cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"status": "Failed",
"startTime": "2026-11-13T07:22:34.538735812Z",
"endTime": "2026-11-13T07:41:41.602384099Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-lv9kw-7q9mn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-lv9kw-7q9mn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
correlation ID: 9af047e6-a985-412b-a5f1-08a259c865d2
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1bcb38aea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-ccvxrx6srd7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"name": "ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"status": "Failed",
"startTime": "2026-11-13T03:38:48.906568999Z",
"endTime": "2026-11-13T03:57:50.268469646Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-x596d-7l8cz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-x596d-7l8cz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
correlation ID: 975c1ede-92c5-42fb-b310-1b83c34bc7da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1bcb38aea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-ccvxrx6srd7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"name": "ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"status": "Failed",
"startTime": "2026-11-13T03:38:48.906568999Z",
"endTime": "2026-11-13T03:57:50.268469646Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-x596d-7l8cz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-x596d-7l8cz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
correlation ID: 975c1ede-92c5-42fb-b310-1b83c34bc7da
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpen...full failure pattern: ERROR CODE: GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 10 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x336c56bc8a0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8q275c9mcvzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/22a945b7-52c3-424e-9fb6-18bc57cfd41f
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/22a945b7-52c3-424e-9fb6-18bc57cfd41f
correlation ID: 64973b7d-5b0c-402d-b88e-6068b1d61261
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x336c56bc8a0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8q275c9mcvzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/22a945b7-52c3-424e-9fb6-18bc57cfd41f
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/22a945b7-52c3-424e-9fb6-18bc57cfd41f
correlation ID: 64973b7d-5b0c-402d-b88e-6068b1d61261
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0xfb96d004160>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-2w95q967z8cb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/287ba809-6380-45aa-b58a-efefac5ecabc
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/287ba809-6380-45aa-b58a-efefac5ecabc
correlation ID: 1817b269-4672-49a3-9ced-3b4328ee8878
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0xfb96d004160>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-2w95q967z8cb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/287ba809-6380-45aa-b58a-efefac5ecabc
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/287ba809-6380-45aa-b58a-efefac5ecabc
correlation ID: 1817b269-4672-49a3-9ced-3b4328ee8878
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x232b0252180>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4v2cqwm888bq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ded81d2e-efec-470b-abd8-a812c874f40c
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ded81d2e-efec-470b-abd8-a812c874f40c
correlation ID: 396a7f9f-95d8-49d9-8906-939e848022d3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x232b0252180>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4v2cqwm888bq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ded81d2e-efec-470b-abd8-a812c874f40c
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ded81d2e-efec-470b-abd8-a812c874f40c
correlation ID: 396a7f9f-95d8-49d9-8906-939e848022d3
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 2 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj"
Unexpected error:
<*fmt.wrapError | 0x9ce8fef8480>:
failed to create NodePool np-hw65wj: failed waiting for nodepool="np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj" in resourcegroup="rg-candidate-4-22-njg7rgmmg8l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d",
"name": "63497251-36b4-416b-a60b-757bca5b1a6d",
"status": "Failed",
"startTime": "2026-11-13T03:59:57.353246771Z",
"endTime": "2026-11-13T04:19:05.626847985Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-hw65wj-h7fm8-hlfd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-hw65wj-h7fm8-hlfd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
correlation ID: f5bd758f-09c6-4157-8de6-ff700a1dc700
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj"
Unexpected error:
<*fmt.wrapError | 0x9ce8fef8480>:
failed to create NodePool np-hw65wj: failed waiting for nodepool="np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj" in resourcegroup="rg-candidate-4-22-njg7rgmmg8l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d",
"name": "63497251-36b4-416b-a60b-757bca5b1a6d",
"status": "Failed",
"startTime": "2026-11-13T03:59:57.353246771Z",
"endTime": "2026-11-13T04:19:05.626847985Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-hw65wj-h7fm8-hlfd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-hw65wj-h7fm8-hlfd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
correlation ID: f5bd758f-09c6-4157-8de6-ff700a1dc700
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5"
Unexpected error:
<*fmt.wrapError | 0x3acf2e617ec0>:
failed to create NodePool np-4smnl5: failed waiting for nodepool="np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5" in resourcegroup="rg-candidate-4-23-bqlkqphqlrhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"name": "d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"status": "Failed",
"startTime": "2026-11-13T04:02:01.972677601Z",
"endTime": "2026-11-13T04:21:05.56135744Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-4smnl5-nm98c-k5msg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-4smnl5-nm98c-k5msg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
correlation ID: fc7ff880-2948-4807-a4fc-dcd90799df98
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5"
Unexpected error:
<*fmt.wrapError | 0x3acf2e617ec0>:
failed to create NodePool np-4smnl5: failed waiting for nodepool="np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5" in resourcegroup="rg-candidate-4-23-bqlkqphqlrhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"name": "d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"status": "Failed",
"startTime": "2026-11-13T04:02:01.972677601Z",
"endTime": "2026-11-13T04:21:05.56135744Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-4smnl5-nm98c-k5msg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-4smnl5-nm98c-k5msg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
correlation ID: fc7ff880-2948-4807-a4fc-dcd90799df98
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | DEV, INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x7f4168e4fc0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4ee21bb8-911f-48f9-95ba-212ff2978055
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4ee21bb8-911f-48f9-95ba-212ff2978055",
"name": "4ee21bb8-911f-48f9-95ba-212ff2978055",
"status": "Failed",
"startTime": "2026-11-18T00:32:24.268028299Z",
"endTime": "2026-11-18T01:01:25.98437898Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4ee21bb8-911f-48f9-95ba-212ff2978055
correlation ID: 37c0466d-4776-48fe-884b-b017ab576eb6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x7f4168e4fc0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4ee21bb8-911f-48f9-95ba-212ff2978055
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4ee21bb8-911f-48f9-95ba-212ff2978055",
"name": "4ee21bb8-911f-48f9-95ba-212ff2978055",
"status": "Failed",
"startTime": "2026-11-18T00:32:24.268028299Z",
"endTime": "2026-11-18T01:01:25.98437898Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4ee21bb8-911f-48f9-95ba-212ff2978055
correlation ID: 37c0466d-4776-48fe-884b-b017ab576eb6
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x29f5eb6eecc0>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-9r7nqlhhcbcf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4",
"name": "dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4",
"status": "Failed",
"startTime": "2026-11-18T11:54:50.797224067Z",
"endTime": "2026-11-18T12:23:50.395523338Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4
correlation ID: 63a31b01-0e2f-4d5f-8d3c-d9cb12d9ea71
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x29f5eb6eecc0>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-9r7nqlhhcbcf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4",
"name": "dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4",
"status": "Failed",
"startTime": "2026-11-18T11:54:50.797224067Z",
"endTime": "2026-11-18T12:23:50.395523338Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/dfda7b2f-86e8-4ba5-9d24-50b8fa2ffda4
correlation ID: 63a31b01-0e2f-4d5f-8d3c-d9cb12d9ea71
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:102]: failed to verify HCP cluster "cilium-cl" is available
Expected success, but got an error:
<*errors.joinError | 0x1e462bf3cba0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1e462c02b5c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1e462c02b5a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1e462bfb01e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1e462b7f8e20>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1e462b7f8e00>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1e462c0860a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:102]: failed to verify HCP cluster "cilium-cl" is available
Expected success, but got an error:
<*errors.joinError | 0x1e462bf3cba0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1e462c02b5c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1e462c02b5a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1e462bfb01e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1e462b7f8e20>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1e462b7f8e00>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1e462c0860a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x2742697ee600>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/445b370d-7341-4972-a0c7-2f20eefc2ab7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/445b370d-7341-4972-a0c7-2f20eefc2ab7",
"name": "445b370d-7341-4972-a0c7-2f20eefc2ab7",
"status": "Failed",
"startTime": "2026-11-18T13:05:34.673486809Z",
"endTime": "2026-11-18T13:34:40.209975764Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/445b370d-7341-4972-a0c7-2f20eefc2ab7
correlation ID: 357df291-0b7d-49ac-9b7e-13e664a8aac1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x2742697ee600>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/445b370d-7341-4972-a0c7-2f20eefc2ab7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/445b370d-7341-4972-a0c7-2f20eefc2ab7",
"name": "445b370d-7341-4972-a0c7-2f20eefc2ab7",
"status": "Failed",
"startTime": "2026-11-18T13:05:34.673486809Z",
"endTime": "2026-11-18T13:34:40.209975764Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/445b370d-7341-4972-a0c7-2f20eefc2ab7
correlation ID: 357df291-0b7d-49ac-9b7e-13e664a8aac1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: AzureMachine status.initialization.provisioned is false; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.46
Unexpected error:
<*fmt.wrapError | 0x298ff8376680>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-hf5grb" in resourcegroup="rg-np-skip-minor-hf5grb-482lzd4v57gn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05",
"name": "a5613375-9443-4d68-9e17-0ff7a2c93d05",
"status": "Failed",
"startTime": "2026-11-13T07:18:53.583036999Z",
"endTime": "2026-11-13T07:38:01.651580211Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-nps-420-422-5r77b-9fj4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-nps-420-422-5r77b-zmkv2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-nps-420-422-5r77b-9fj4g: NotReady: AzureMachine status.initialization.provisioned is false\nMachine cluster-nps-420-422-5r77b-zmkv2: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
correlation ID: 9bc3ecfd-be4c-41ac-bac1-303b965f5bd5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.46
Unexpected error:
<*fmt.wrapError | 0x298ff8376680>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-hf5grb" in resourcegroup="rg-np-skip-minor-hf5grb-482lzd4v57gn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05",
"name": "a5613375-9443-4d68-9e17-0ff7a2c93d05",
"status": "Failed",
"startTime": "2026-11-13T07:18:53.583036999Z",
"endTime": "2026-11-13T07:38:01.651580211Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-nps-420-422-5r77b-9fj4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-nps-420-422-5r77b-zmkv2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-nps-420-422-5r77b-9fj4g: NotReady: AzureMachine status.initialization.provisioned is false\nMachine cluster-nps-420-422-5r77b-zmkv2: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
correlation ID: 9bc3ecfd-be4c-41ac-bac1-303b965f5bd5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x164258d863c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xs6vdbh9ddbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01",
"name": "7558c7b9-327a-43c2-8293-9ac1255cbb01",
"status": "Failed",
"startTime": "2026-11-13T04:30:51.785572468Z",
"endTime": "2026-11-13T04:50:00.485199208Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
correlation ID: e0fc903a-bf7b-4e5b-a7e3-14066f61f36f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x164258d863c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xs6vdbh9ddbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01",
"name": "7558c7b9-327a-43c2-8293-9ac1255cbb01",
"status": "Failed",
"startTime": "2026-11-13T04:30:51.785572468Z",
"endTime": "2026-11-13T04:50:00.485199208Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
correlation ID: e0fc903a-bf7b-4e5b-a7e3-14066f61f36f
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Addi...full failure pattern: ERROR CODE: DeadlineExceeded; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum) New Limit R... message truncated AutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2).; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x15f0f6f7f240>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mbb55929zcrq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e36e2c62-e78b-4fa7-b4bc-5f660647c0d9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e36e2c62-e78b-4fa7-b4bc-5f660647c0d9",
"name": "e36e2c62-e78b-4fa7-b4bc-5f660647c0d9",
"status": "Failed",
"startTime": "2026-11-17T23:21:07.671812556Z",
"endTime": "2026-11-17T23:40:11.56037759Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-h8t9t-8zdlp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-h8t9t-m6f4c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-h8t9t-8zdlp: Failed: virtualmachine failed to create or update. err: failed to create or update resource autoscaling-cluster-mbb55929zcrq--managed/cluster-np-1-h8t9t-8zdlp (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-mbb55929zcrq--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-h8t9t-8zdlp\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum) New Limit R... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e36e2c62-e78b-4fa7-b4bc-5f660647c0d9
correlation ID: 4632c86c-5aa5-4a2c-8b57-549734d97cfa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x15f0f6f7f240>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mbb55929zcrq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e36e2c62-e78b-4fa7-b4bc-5f660647c0d9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e36e2c62-e78b-4fa7-b4bc-5f660647c0d9",
"name": "e36e2c62-e78b-4fa7-b4bc-5f660647c0d9",
"status": "Failed",
"startTime": "2026-11-17T23:21:07.671812556Z",
"endTime": "2026-11-17T23:40:11.56037759Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-h8t9t-8zdlp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-h8t9t-m6f4c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-h8t9t-8zdlp: Failed: virtualmachine failed to create or update. err: failed to create or update resource autoscaling-cluster-mbb55929zcrq--managed/cluster-np-1-h8t9t-8zdlp (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-mbb55929zcrq--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-h8t9t-8zdlp\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum) New Limit R... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e36e2c62-e78b-4fa7-b4bc-5f660647c0d9
correlation ID: 4632c86c-5aa5-4a2c-8b57-549734d97cfa
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. ...full failure pattern: ERROR CODE: DeadlineExceeded; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastasia, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum)... message truncated AutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2).; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
<*fmt.wrapError | 0xd0b821828e0>:
failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-4lxhkw8scjfd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/51b0f828-231f-453a-9a2d-5ed36c437058
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/51b0f828-231f-453a-9a2d-5ed36c437058",
"name": "51b0f828-231f-453a-9a2d-5ed36c437058",
"status": "Failed",
"startTime": "2026-11-18T13:03:59.023514015Z",
"endTime": "2026-11-18T13:23:00.361585763Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-ea-np-1-wjkkk-76xd6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-ea-np-1-wjkkk-76xd6: Failed: virtualmachine failed to create or update. err: failed to create or update resource external-auth-cluster-4lxhkw8scjfd--managed/cluster-ea-np-1-wjkkk-76xd6 (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4lxhkw8scjfd--managed/providers/Microsoft.Compute/virtualMachines/cluster-ea-np-1-wjkkk-76xd6\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastasia, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum)... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/51b0f828-231f-453a-9a2d-5ed36c437058
correlation ID: 79cf33bc-6b74-47af-89f3-1ebb42494ea2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
<*fmt.wrapError | 0xd0b821828e0>:
failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-4lxhkw8scjfd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/51b0f828-231f-453a-9a2d-5ed36c437058
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/51b0f828-231f-453a-9a2d-5ed36c437058",
"name": "51b0f828-231f-453a-9a2d-5ed36c437058",
"status": "Failed",
"startTime": "2026-11-18T13:03:59.023514015Z",
"endTime": "2026-11-18T13:23:00.361585763Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-ea-np-1-wjkkk-76xd6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-ea-np-1-wjkkk-76xd6: Failed: virtualmachine failed to create or update. err: failed to create or update resource external-auth-cluster-4lxhkw8scjfd--managed/cluster-ea-np-1-wjkkk-76xd6 (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4lxhkw8scjfd--managed/providers/Microsoft.Compute/virtualMachines/cluster-ea-np-1-wjkkk-76xd6\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastasia, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum)... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperationStatuses/51b0f828-231f-453a-9a2d-5ed36c437058
correlation ID: 79cf33bc-6b74-47af-89f3-1ebb42494ea2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. ...full failure pattern: ERROR CODE: DeadlineExceeded; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Min... message truncated AutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2).; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
<*fmt.wrapError | 0xeb5482df7c0>:
failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-th9q266qt25p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/feb6c392-6233-41de-ba11-c17914050ad8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/feb6c392-6233-41de-ba11-c17914050ad8",
"name": "feb6c392-6233-41de-ba11-c17914050ad8",
"status": "Failed",
"startTime": "2026-11-18T02:51:47.079879028Z",
"endTime": "2026-11-18T03:10:52.47747104Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-ea-np-1-znqtd-28pn5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-ea-np-1-znqtd-28pn5: Failed: virtualmachine failed to create or update. err: failed to create or update resource external-auth-cluster-th9q266qt25p--managed/cluster-ea-np-1-znqtd-28pn5 (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-th9q266qt25p--managed/providers/Microsoft.Compute/virtualMachines/cluster-ea-np-1-znqtd-28pn5\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Min... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/feb6c392-6233-41de-ba11-c17914050ad8
correlation ID: 2801c93b-9819-4120-81e1-d6e9fcde271f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
<*fmt.wrapError | 0xeb5482df7c0>:
failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-th9q266qt25p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/feb6c392-6233-41de-ba11-c17914050ad8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/feb6c392-6233-41de-ba11-c17914050ad8",
"name": "feb6c392-6233-41de-ba11-c17914050ad8",
"status": "Failed",
"startTime": "2026-11-18T02:51:47.079879028Z",
"endTime": "2026-11-18T03:10:52.47747104Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-ea-np-1-znqtd-28pn5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-ea-np-1-znqtd-28pn5: Failed: virtualmachine failed to create or update. err: failed to create or update resource external-auth-cluster-th9q266qt25p--managed/cluster-ea-np-1-znqtd-28pn5 (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-th9q266qt25p--managed/providers/Microsoft.Compute/virtualMachines/cluster-ea-np-1-znqtd-28pn5\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"OperationNotAllowed\",\n \"message\": \"Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: swedencentral, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Min... message truncated\n\nAutoRepair is enabled; the MachineHealthCheck will attempt to replace unhealthy machines unless the number of unhealthy machines exceeds maxUnhealthy (default 2)."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/feb6c392-6233-41de-ba11-c17914050ad8
correlation ID: 2801c93b-9819-4120-81e1-d6e9fcde271f
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x132e3634f980>:
failed to create HCP cluster ea-list9fmcxd: failed waiting for cluster="ea-list9fmcxd" in resourcegroup="ea-list-lgcwmrk5lcqm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4e8ca130-a83d-4b69-962f-d5637a65184d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4e8ca130-a83d-4b69-962f-d5637a65184d",
"name": "4e8ca130-a83d-4b69-962f-d5637a65184d",
"status": "Failed",
"startTime": "2026-11-18T00:06:02.917533725Z",
"endTime": "2026-11-18T00:35:02.521374697Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4e8ca130-a83d-4b69-962f-d5637a65184d
correlation ID: 5b531f48-5847-40e0-ad30-35c2e97cb633
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x132e3634f980>:
failed to create HCP cluster ea-list9fmcxd: failed waiting for cluster="ea-list9fmcxd" in resourcegroup="ea-list-lgcwmrk5lcqm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4e8ca130-a83d-4b69-962f-d5637a65184d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4e8ca130-a83d-4b69-962f-d5637a65184d",
"name": "4e8ca130-a83d-4b69-962f-d5637a65184d",
"status": "Failed",
"startTime": "2026-11-18T00:06:02.917533725Z",
"endTime": "2026-11-18T00:35:02.521374697Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/4e8ca130-a83d-4b69-962f-d5637a65184d
correlation ID: 5b531f48-5847-40e0-ad30-35c2e97cb633
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x3707c1a22b60>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-qshnd2g9qq8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f",
"name": "dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f",
"status": "Failed",
"startTime": "2026-11-18T00:06:00.91339116Z",
"endTime": "2026-11-18T00:35:02.461317054Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f
correlation ID: 05dc3cc2-fee4-47dc-b745-fcc3d920b2f8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x3707c1a22b60>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-qshnd2g9qq8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f",
"name": "dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f",
"status": "Failed",
"startTime": "2026-11-18T00:06:00.91339116Z",
"endTime": "2026-11-18T00:35:02.461317054Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/dc5e28a9-7aa3-40e9-a64f-5c04a4f29c3f
correlation ID: 05dc3cc2-fee4-47dc-b745-fcc3d920b2f8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 1 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x3c742d844860>:
failed to create HCP cluster ea-listhzmj8j: failed waiting for cluster="ea-listhzmj8j" in resourcegroup="ea-list-77xrkhvsxtg2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e86df344-528f-4ede-a688-3f6b61315b21
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e86df344-528f-4ede-a688-3f6b61315b21",
"name": "e86df344-528f-4ede-a688-3f6b61315b21",
"status": "Failed",
"startTime": "2026-11-18T02:45:52.083606789Z",
"endTime": "2026-11-18T03:14:52.462344689Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 2 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e86df344-528f-4ede-a688-3f6b61315b21
correlation ID: a225d2e0-6b1d-403e-b826-03d60a0f770c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x3c742d844860>:
failed to create HCP cluster ea-listhzmj8j: failed waiting for cluster="ea-listhzmj8j" in resourcegroup="ea-list-77xrkhvsxtg2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e86df344-528f-4ede-a688-3f6b61315b21
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e86df344-528f-4ede-a688-3f6b61315b21",
"name": "e86df344-528f-4ede-a688-3f6b61315b21",
"status": "Failed",
"startTime": "2026-11-18T02:45:52.083606789Z",
"endTime": "2026-11-18T03:14:52.462344689Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 2 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperationStatuses/e86df344-528f-4ede-a688-3f6b61315b21
correlation ID: a225d2e0-6b1d-403e-b826-03d60a0f770c
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x691f5c08960>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7ndhwr7f8prm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761",
"name": "3852afe7-3cef-47a2-b8aa-4d84a9736761",
"status": "Failed",
"startTime": "2026-11-13T06:16:07.058086055Z",
"endTime": "2026-11-13T06:35:15.725926802Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-g2fdm-7l2bn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-cilium-np-g2fdm-7l2bn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
correlation ID: f6bc5e83-47cf-48b3-a570-66ef10fd1d63
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x691f5c08960>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7ndhwr7f8prm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761",
"name": "3852afe7-3cef-47a2-b8aa-4d84a9736761",
"status": "Failed",
"startTime": "2026-11-13T06:16:07.058086055Z",
"endTime": "2026-11-13T06:35:15.725926802Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-g2fdm-7l2bn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-cilium-np-g2fdm-7l2bn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
correlation ID: f6bc5e83-47cf-48b3-a570-66ef10fd1d63
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:192]: failed to create node pool "np-1" for fully private cluster with KV "full-priv-kv"
Unexpected error:
<*fmt.wrapError | 0x399db18e1c20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "full-priv-kv" in resourcegroup="full-priv-kv-xts7895q2w9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5",
"name": "6ee03316-feff-4093-9246-bef8a3716fd5",
"status": "Failed",
"startTime": "2026-11-13T06:16:32.006714258Z",
"endTime": "2026-11-13T06:35:35.830707205Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-whczx-4dxn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-whczx-4dxn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
correlation ID: 003842c9-adb0-44c1-b432-98cc60f79fe8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:192]: failed to create node pool "np-1" for fully private cluster with KV "full-priv-kv"
Unexpected error:
<*fmt.wrapError | 0x399db18e1c20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "full-priv-kv" in resourcegroup="full-priv-kv-xts7895q2w9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5",
"name": "6ee03316-feff-4093-9246-bef8a3716fd5",
"status": "Failed",
"startTime": "2026-11-13T06:16:32.006714258Z",
"endTime": "2026-11-13T06:35:35.830707205Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-whczx-4dxn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-whczx-4dxn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
correlation ID: 003842c9-adb0-44c1-b432-98cc60f79fe8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc"
Unexpected error:
<*fmt.wrapError | 0x4de899b99e0>:
failed to create NodePool np-qzz6xc: failed waiting for nodepool="np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc" in resourcegroup="rg-candidate-4-21-qzgwvl6kwrmt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"name": "f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"status": "Failed",
"startTime": "2026-11-13T06:42:31.271421653Z",
"endTime": "2026-11-13T07:01:35.883206031Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-qzz6xc-57t85-pxcdm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-qzz6xc-57t85-pxcdm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
correlation ID: 99f225d1-e26c-43e5-9f87-e0f4765d3804
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc"
Unexpected error:
<*fmt.wrapError | 0x4de899b99e0>:
failed to create NodePool np-qzz6xc: failed waiting for nodepool="np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc" in resourcegroup="rg-candidate-4-21-qzgwvl6kwrmt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"name": "f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"status": "Failed",
"startTime": "2026-11-13T06:42:31.271421653Z",
"endTime": "2026-11-13T07:01:35.883206031Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-qzz6xc-57t85-pxcdm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-qzz6xc-57t85-pxcdm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
correlation ID: 99f225d1-e26c-43e5-9f87-e0f4765d3804
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-<id> to exist Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.46
Unexpected error:
<*fmt.wrapError | 0xbf3c7f945e0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-2klx7r" in resourcegroup="rg-np-version-upgrade-2klx7r-qhhp8zd44mhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"name": "2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"status": "Failed",
"startTime": "2026-11-13T04:32:00.613107854Z",
"endTime": "2026-11-13T04:51:00.48916791Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
correlation ID: 628c311f-b207-4ece-9d6b-da4a010d2a2c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.46
Unexpected error:
<*fmt.wrapError | 0xbf3c7f945e0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-2klx7r" in resourcegroup="rg-np-version-upgrade-2klx7r-qhhp8zd44mhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"name": "2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"status": "Failed",
"startTime": "2026-11-13T04:32:00.613107854Z",
"endTime": "2026-11-13T04:51:00.48916791Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
correlation ID: 628c311f-b207-4ece-9d6b-da4a010d2a2c
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xc4ddaff8bb8>:
VerifyNodesReady failed: there are not ready nodes: cluster-agg-np-a-dz7h5-wdm8j (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-11-13T06:34:17Z)
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v94sn66nsnfk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5",
"name": "427fbf99-b011-448a-8853-e0e336f802e5",
"status": "Failed",
"startTime": "2026-11-13T06:16:51.858643855Z",
"endTime": "2026-11-13T06:35:55.74861517Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-agg-np-a-dz7h5-wdm8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-agg-np-a-dz7h5-wdm8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
correlation ID: 3d6b880e-0708-43ee-96e6-2869a452519b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xc4ddaff8bb8>:
VerifyNodesReady failed: there are not ready nodes: cluster-agg-np-a-dz7h5-wdm8j (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-11-13T06:34:17Z)
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v94sn66nsnfk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5",
"name": "427fbf99-b011-448a-8853-e0e336f802e5",
"status": "Failed",
"startTime": "2026-11-13T06:16:51.858643855Z",
"endTime": "2026-11-13T06:35:55.74861517Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-agg-np-a-dz7h5-wdm8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-agg-np-a-dz7h5-wdm8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
correlation ID: 3d6b880e-0708-43ee-96e6-2869a452519b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AlreadyExists; detail message [AlreadyExists (Conflict)] A key with name 'etcd-data-kms-encryption-key' already exists in this key vault.full failure pattern: ERROR CODE: DeploymentFailed; detail code AlreadyExists; detail message [AlreadyExists (Conflict)] A key with name 'etcd-data-kms-encryption-key' already exists in this key vault. Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:268]: failed to create cluster customer resources
Unexpected error:
<*fmt.wrapError | 0x1b69e01376c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-shoebox-hcp-cluster-5gm6gd" in resourcegroup="shoebox-cluster-2nrhtgfcxbvs" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/shoebox-cluster-2nrhtgfcxbvs/providers/Microsoft.Resources/deployments/customer-infra-shoebox-hcp-cluster-5gm6gd/operationStatuses/08584066451748390759
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 7d665c7c-9c23-405d-89c9-60eeff90ff75",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"AlreadyExists\",\r\n \"message\": \"[AlreadyExists (Conflict)] A key with name 'etcd-data-kms-encryption-key' already exists in this key vault.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/shoebox-cluster-2nrhtgfcxbvs/providers/Microsoft.Resources/deployments/customer-infra-shoebox-hcp-cluster-5gm6gd/operationStatuses/08584066451748390759
correlation ID: 2670328f-8977-435b-afd8-67b41c3e759b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:268]: failed to create cluster customer resources
Unexpected error:
<*fmt.wrapError | 0x1b69e01376c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-shoebox-hcp-cluster-5gm6gd" in resourcegroup="shoebox-cluster-2nrhtgfcxbvs" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/shoebox-cluster-2nrhtgfcxbvs/providers/Microsoft.Resources/deployments/customer-infra-shoebox-hcp-cluster-5gm6gd/operationStatuses/08584066451748390759
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 7d665c7c-9c23-405d-89c9-60eeff90ff75",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"AlreadyExists\",\r\n \"message\": \"[AlreadyExists (Conflict)] A key with name 'etcd-data-kms-encryption-key' already exists in this key vault.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/shoebox-cluster-2nrhtgfcxbvs/providers/Microsoft.Resources/deployments/customer-infra-shoebox-hcp-cluster-5gm6gd/operationStatuses/08584066451748390759
correlation ID: 2670328f-8977-435b-afd8-67b41c3e759b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code ConflictError; detail message A conflict occurred that prevented the operation from completing. The operation failed because the Key Vaul...full failure pattern: ERROR CODE: DeploymentFailed; detail code ConflictError; detail message A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: <url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:154]: failed to create customer resources for third cluster "basic-hcp-cluster3"
Unexpected error:
<*fmt.wrapError | 0xfef0a54cd20>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster3-wrxpnc" in resourcegroup="customer-rg-nxkg9nqd5rmk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nxkg9nqd5rmk/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster3-wrxpnc/operationStatuses/08584066976503511107
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 578a25f6-51a2-4c61-a78e-2caf0c1c10a9",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"ConflictError\",\r\n \"message\": \"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nxkg9nqd5rmk/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster3-wrxpnc/operationStatuses/08584066976503511107
correlation ID: 29db9e2d-e8b7-42a6-a93a-2bdd20756033
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:154]: failed to create customer resources for third cluster "basic-hcp-cluster3"
Unexpected error:
<*fmt.wrapError | 0xfef0a54cd20>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster3-wrxpnc" in resourcegroup="customer-rg-nxkg9nqd5rmk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nxkg9nqd5rmk/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster3-wrxpnc/operationStatuses/08584066976503511107
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 578a25f6-51a2-4c61-a78e-2caf0c1c10a9",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"ConflictError\",\r\n \"message\": \"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nxkg9nqd5rmk/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster3-wrxpnc/operationStatuses/08584066976503511107
correlation ID: 29db9e2d-e8b7-42a6-a93a-2bdd20756033
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code InternalServerErrorfull failure pattern: ERROR CODE: DeploymentFailed; detail code InternalServerError Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:78]: failed to create customer resources for cilium CNI cluster
Unexpected error:
<*fmt.wrapError | 0xb8f673a2b00>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cilium-cl-l8n7vs" in resourcegroup="cni-cilium-qhtsvpmkrk26" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/cni-cilium-qhtsvpmkrk26/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-l8n7vs/operationStatuses/08584067024950129218
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 18211a48-e456-4bc3-8b16-15591a2fd217",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20261117T083417Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0c71d6bc-583c-4464-8638-006e74fd3ff1', request correlation id '18211a48-e456-4bc3-8b16-15591a2fd217'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/cni-cilium-qhtsvpmkrk26/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-l8n7vs/operationStatuses/08584067024950129218
correlation ID: 42ced3db-fe31-425d-aaac-155355fd9643
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:78]: failed to create customer resources for cilium CNI cluster
Unexpected error:
<*fmt.wrapError | 0xb8f673a2b00>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cilium-cl-l8n7vs" in resourcegroup="cni-cilium-qhtsvpmkrk26" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/cni-cilium-qhtsvpmkrk26/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-l8n7vs/operationStatuses/08584067024950129218
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 18211a48-e456-4bc3-8b16-15591a2fd217",
"details": [
{
"code": "InternalServerError",
"message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20261117T083417Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0c71d6bc-583c-4464-8638-006e74fd3ff1', request correlation id '18211a48-e456-4bc3-8b16-15591a2fd217'.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/cni-cilium-qhtsvpmkrk26/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-l8n7vs/operationStatuses/08584067024950129218
correlation ID: 42ced3db-fe31-425d-aaac-155355fd9643
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail code GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time per...full failure pattern: ERROR CODE: InvalidRequestContent; detail code GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:145]: failed to get admin REST config for cluster "cp-ystream-upgrade-5-1-c9f7wn"
Unexpected error:
<*fmt.wrapErrors | 0x3c94e48e2450>:
20240610 credential request failed: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-x78mcnhnwhwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-c9f7wn/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Required value",
"target": "certificateSigningRequest"
}
}
--------------------------------------------------------------------------------
; 2026-09-01-preview fallback failed: failed waiting for hcpCluster="cp-ystream-upgrade-5-1-c9f7wn" in resourcegroup="rg-cp-ystream-upgrade-5-1-x78mcnhnwhwq" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3fe38471-2bc1-4424-b744-e25a3128b5fb
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-x78mcnhnwhwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-c9f7wn/requestAdminCredential
correlation ID: 3ef5d25c-72ff-43c0-986a-3c682572ea94
client request ID: unavailable
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3fe38471-2bc1-4424-b744-e25a3128b5fb
correlation ID: 2a4e5c4e-d06c-41dd-90b1-09a9fddd82a1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:145]: failed to get admin REST config for cluster "cp-ystream-upgrade-5-1-c9f7wn"
Unexpected error:
<*fmt.wrapErrors | 0x3c94e48e2450>:
20240610 credential request failed: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-x78mcnhnwhwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-c9f7wn/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Required value",
"target": "certificateSigningRequest"
}
}
--------------------------------------------------------------------------------
; 2026-09-01-preview fallback failed: failed waiting for hcpCluster="cp-ystream-upgrade-5-1-c9f7wn" in resourcegroup="rg-cp-ystream-upgrade-5-1-x78mcnhnwhwq" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3fe38471-2bc1-4424-b744-e25a3128b5fb
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-x78mcnhnwhwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-c9f7wn/requestAdminCredential
correlation ID: 3ef5d25c-72ff-43c0-986a-3c682572ea94
client request ID: unavailable
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3fe38471-2bc1-4424-b744-e25a3128b5fb
correlation ID: 2a4e5c4e-d06c-41dd-90b1-09a9fddd82a1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationRequired; provider Microsoft.Computefull failure pattern: ERROR CODE: LocationRequired; provider Microsoft.Compute Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-20-67w8lw"
Unexpected error:
<*errors.joinError | 0x80cae74a300>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-qgtvc/e2e-csi-pvc phase=Bound volumeName=pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 storageClass=managed-csi
[pod] e2e-csi-disk-qgtvc/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodReadyToStartContainers status=False reason= message=
[pod] condition type=Initialized status=True reason= message=
[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=PodScheduled status=True reason= message=
[pod] container volume-check ready=false restartCount=0
[pod] waiting reason=ContainerCreating message=
[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-qgtvc/e2e-csi-pod to cluster-np-67w8lw-ns2jx-fhmn9 (reason=Scheduled, count=0, age=9m54s)
[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327" : rpc error: code = Internal desc = update instance "cluster-np-67w8lw-ns2jx-fhmn9" failed with PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p46rhcdq94l5-candidate-67w8lw--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-67w8lw-ns2jx-fhmn9
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationRequired
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationRequired",
"message": "The location property is required for this definition."
}
}
--------------------------------------------------------------------------------
(reason=FailedAttachVolume, count=12, age=26s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-qgtvc/e2e-csi-pvc" (reason=Provisioning, count=1, age=10m0s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m55s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 (reason=ProvisioningSucceeded, count=1, age=9m54s)
{
errs: [
<*fmt.wrapError | 0x80cac8aeaa0>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-qgtvc/e2e-csi-pvc phase=Bound volumeName=pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 storageClass=managed-csi\n[pod] e2e-csi-disk-qgtvc/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-qgtvc/e2e-csi-pod to cluster-np-67w8lw-ns2jx-fhmn9 (reason=Scheduled, count=0, age=9m54s)\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327\" : rpc error: code = Internal desc = update instance \"cluster-np-67w8lw-ns2jx-fhmn9\" failed with PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p46rhcdq94l5-candidate-67w8lw--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-67w8lw-ns2jx-fhmn9\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: LocationRequired\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LocationRequired\",\n \"message\": \"The location property is required for this definition.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=FailedAttachVolume, count=12, age=26s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-qgtvc/e2e-csi-pvc\" (reason=Provisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m55s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 (reason=ProvisioningSucceeded, count=1, age=9m54s)",
err: <*fmt.wrapError | 0x80cac8ae960>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-qgtvc/e2e-csi-pvc phase=Bound volumeName=pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 storageClass=managed-csi\n[pod] e2e-csi-disk-qgtvc/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-qgtvc/e2e-csi-pod to cluster-np-67w8lw-ns2jx-fhmn9 (reason=Scheduled, count=0, age=9m54s)\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327\" : rpc error: code = Internal desc = update instance \"cluster-np-67w8lw-ns2jx-fhmn9\" failed with PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p46rhcdq94l5-candidate-67w8lw--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-67w8lw-ns2jx-fhmn9\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: LocationRequired\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LocationRequired\",\n \"message\": \"The location property is required for this definition.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=FailedAttachVolume, count=12, age=26s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-qgtvc/e2e-csi-pvc\" (reason=Provisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m55s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 (reason=ProvisioningSucceeded, count=1, age=9m54s)",
err: <*errors.errorString | 0x80cacb4e4b0>{
s: "pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-20-67w8lw"
Unexpected error:
<*errors.joinError | 0x80cae74a300>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-qgtvc/e2e-csi-pvc phase=Bound volumeName=pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 storageClass=managed-csi
[pod] e2e-csi-disk-qgtvc/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodReadyToStartContainers status=False reason= message=
[pod] condition type=Initialized status=True reason= message=
[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=PodScheduled status=True reason= message=
[pod] container volume-check ready=false restartCount=0
[pod] waiting reason=ContainerCreating message=
[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-qgtvc/e2e-csi-pod to cluster-np-67w8lw-ns2jx-fhmn9 (reason=Scheduled, count=0, age=9m54s)
[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327" : rpc error: code = Internal desc = update instance "cluster-np-67w8lw-ns2jx-fhmn9" failed with PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p46rhcdq94l5-candidate-67w8lw--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-67w8lw-ns2jx-fhmn9
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationRequired
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationRequired",
"message": "The location property is required for this definition."
}
}
--------------------------------------------------------------------------------
(reason=FailedAttachVolume, count=12, age=26s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-qgtvc/e2e-csi-pvc" (reason=Provisioning, count=1, age=10m0s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m55s)
[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 (reason=ProvisioningSucceeded, count=1, age=9m54s)
{
errs: [
<*fmt.wrapError | 0x80cac8aeaa0>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-qgtvc/e2e-csi-pvc phase=Bound volumeName=pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 storageClass=managed-csi\n[pod] e2e-csi-disk-qgtvc/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-qgtvc/e2e-csi-pod to cluster-np-67w8lw-ns2jx-fhmn9 (reason=Scheduled, count=0, age=9m54s)\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327\" : rpc error: code = Internal desc = update instance \"cluster-np-67w8lw-ns2jx-fhmn9\" failed with PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p46rhcdq94l5-candidate-67w8lw--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-67w8lw-ns2jx-fhmn9\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: LocationRequired\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LocationRequired\",\n \"message\": \"The location property is required for this definition.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=FailedAttachVolume, count=12, age=26s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-qgtvc/e2e-csi-pvc\" (reason=Provisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m55s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 (reason=ProvisioningSucceeded, count=1, age=9m54s)",
err: <*fmt.wrapError | 0x80cac8ae960>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-qgtvc/e2e-csi-pvc phase=Bound volumeName=pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 storageClass=managed-csi\n[pod] e2e-csi-disk-qgtvc/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-qgtvc/e2e-csi-pod to cluster-np-67w8lw-ns2jx-fhmn9 (reason=Scheduled, count=0, age=9m54s)\n[events] e2e-csi-disk-qgtvc Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327\" : rpc error: code = Internal desc = update instance \"cluster-np-67w8lw-ns2jx-fhmn9\" failed with PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p46rhcdq94l5-candidate-67w8lw--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-67w8lw-ns2jx-fhmn9\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: LocationRequired\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LocationRequired\",\n \"message\": \"The location property is required for this definition.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=FailedAttachVolume, count=12, age=26s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-qgtvc/e2e-csi-pvc\" (reason=Provisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m55s)\n[events] e2e-csi-disk-qgtvc PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-1a0a0e66-7ea6-4fc4-bf57-8eddbf96d327 (reason=ProvisioningSucceeded, count=1, age=9m54s)",
err: <*errors.errorString | 0x80cacb4e4b0>{
s: "pod e2e-csi-disk-qgtvc/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP failed (ARM HTTP status 504); Dex namespace="console-jfdjrsx"; Deployment/dex condition="Availab...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP failed (ARM HTTP status 504); Dex namespace="console-jfdjrsx"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:253]: failed to set up Console login fixture for cluster "cluster-candidate-4-21-q8tjfn"
Unexpected error:
<*fmt.wrapError | 0x13d6c90201e0>:
console fixture namespace "console-jfdjrsx": configure Console ExternalAuth through RP failed (ARM HTTP status 504); Dex namespace="console-jfdjrsx"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-rtvr4 condition="ContainersReady" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="Initialized" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="PodScheduled" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="Ready" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 container="dex" ready=true restarts=0; Pod/dex-676f8d46b9-rtvr4 phase="Running" reason=""
{
msg: "console fixture namespace \"console-jfdjrsx\": configure Console ExternalAuth through RP failed (ARM HTTP status 504); Dex namespace=\"console-jfdjrsx\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-rtvr4 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 container=\"dex\" ready=true restarts=0; Pod/dex-676f8d46b9-rtvr4 phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x13d6caa82a30>{
s: "configure Console ExternalAuth through RP failed (ARM HTTP status 504)",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:253]: failed to set up Console login fixture for cluster "cluster-candidate-4-21-q8tjfn"
Unexpected error:
<*fmt.wrapError | 0x13d6c90201e0>:
console fixture namespace "console-jfdjrsx": configure Console ExternalAuth through RP failed (ARM HTTP status 504); Dex namespace="console-jfdjrsx"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-rtvr4 condition="ContainersReady" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="Initialized" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="PodScheduled" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 condition="Ready" status="True" reason=""; Pod/dex-676f8d46b9-rtvr4 container="dex" ready=true restarts=0; Pod/dex-676f8d46b9-rtvr4 phase="Running" reason=""
{
msg: "console fixture namespace \"console-jfdjrsx\": configure Console ExternalAuth through RP failed (ARM HTTP status 504); Dex namespace=\"console-jfdjrsx\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-rtvr4 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-rtvr4 container=\"dex\" ready=true restarts=0; Pod/dex-676f8d46b9-rtvr4 phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x13d6caa82a30>{
s: "configure Console ExternalAuth through RP failed (ARM HTTP status 504)",
},
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
selector "<selector>" matched no usable VM size in eastasia: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or ...full failure pattern: selector "<selector>" matched no usable VM size in eastasia: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 1 · Nov 18: 0 · Nov 19: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:92]: failed to find an ARM64-capable VM size; check VM SKU restrictions/quota for the test subscription in eastasia
Unexpected error:
<*fmt.wrapError | 0x2e613489300>:
selector "arm64-worker" matched no usable VM size in eastasia: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"arm64-worker\" matched no usable VM size in eastasia: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x17329c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:92]: failed to find an ARM64-capable VM size; check VM SKU restrictions/quota for the test subscription in eastasia
Unexpected error:
<*fmt.wrapError | 0x2e613489300>:
selector "arm64-worker" matched no usable VM size in eastasia: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"arm64-worker\" matched no usable VM size in eastasia: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x17329c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||